It's been discussed for a long time, and the related proposals were heavily upvoted, including various older proposals.
As I understand it, part of the reason it took a while is that the core Go team was generally of the opinion that doing user-facing SIMD APIs the right way was to design a high-level, cross-platform API that would stand the test of time, and that was then punted a few times given its complexity and need to do other things.
Part of what helped the current approach take off was switching to a philosophy of designing a lower-level architecture-dependent API first (the 'simd/archsimd' package), and then later doing a higher-level portable API (the 'simd' package, which is topic of this blog post).
That two-level approach I think also gave some additional freedom for the design and implementation of the friendlier / high-level 'simd' package, including because the lower-level 'simd/archsimd' package is available for people who need or want to drop down.
> I have never seen real Go code (i.e. not code written purposefully to be exploitable) that was exploitable due to a data race.
And from tptacek in that same discussion [2]:
> The fact is that Go doesn't admit memory corruption vulnerabilities, and the way you know that is the fact that there are practically zero exploits for memory corruption vulnerabilities targeting pure Go programs, despite the popularity of the language.
(The gist: memory safety is a term of art coined by security practitioners. Go, Python, Rust, Java, others: memory safe. C/C++: memory unsafe. Periodically, people in different slices of industry or academia come up with new definitions of memory safety that declare Rust or Go or other languages to be memory unsafe, but that is not by the broadly accepted definition across industry.)
Since we are not at some point in the future where that correct compiler exists and there is only one official compiler, the distinction you make is practically meaningless!
I mean… no? It matters whether something is a part of the language or not, because it matters if you can write code relying on this behavior. Since this is a compiler bug, you cannot - the code will stop compiling the moment the bug is fixed.
There are no known instances of this bug being encountered in the wild, and if you look into it, you will see how extremely unlikely such code is.
> Isn't ten years enough to call something a feature of the language rather than a bug?
I suppose it depends on who is doing the classifying? From the developer's standpoint I'd imagine intent is all that matters: a bug is something that does not match developer intent and that is (eventually) expected to be changed to match the intent, while a feature is something that does match developer intent regardless of how old/new it is. From a user's standpoint I'd imagine it's a combination of developer intent and the user's reliance on said behavior, but IIRC in this particular case there's no known non-demo code that has organically run into this particular bug so there's little weight in favor of calling the bug a feature despite the devs' stance.
Also as GP said I think one needs to be careful to distinguish between the compiler and the language. IIRC the devs have known for basically this entire time exactly in what manner the Rust compiler fail to implement the rules of Rust the language, but a general fix has been blocked on long-running projects that have only recently been approaching the finish line [1].
I define undefined behaviour as a bug in C++. Now C++ is memory-safe!
btw, it's not actually that hard to write correct code in C++, easier than in C because you have all the container types. The problem is that nothing will tell you when you write incorrect code - there's no guarantee.
UB is part of the C++ standard. Surely you can understand the difference between the C++ standard and bugs in compilers implementing the C++ standard. This is that.
Rust does not have an ISO standard, but it does have a language design, and if you knew the first thing about cve-rs (including what’s on its own Github page), you would know that this is an extremely confirmed soundness bug.
The cve-rs repo is not meant to be the toxic gotcha aimed at Rust language maintainers you seem to think it is. It’s a repro case.
> I define undefined behaviour as a bug in C++. Now C++ is memory-safe!
I mean, sure, insofar as such a thing would also imply that a) the standard would need quite a bit of cleanup/clarification work to not contradict your definition, and b) the main optimizing C++ compilers are miscompiling code, analogous to how cve-rs is a rustc miscompilation rather than an issue with Rust itself.
(Fil-C might be an interesting exception here, though IIRC its definition of memory safety is slightly different)
Well, you're right about one thing: the fact that I've spent my career in software security doesn't make me "broad consideration". The cites I give on what "memory safe" means, though, do.
My argument has never been "memory safe means what I say it does because I say so", but I get how that's a much more convenient argument to knock down than the ISRG site built specifically to talk about this.
ISRG is wrong too, definition-wise. Go is not memory safe. It is much safer, and I wouldn’t fault you for taking a C codebase and porting it to Go to avoid memory safety problems, but that does not make it memory safe in the same way Rust et al are memory safe. This is the same way that MTE does not thwart all memory corruption but it stops a lot of them. I accept your premise that Go has brought memory safety over the line to where it is apparently easier to find logic bugs than exploit memory corruption, which is laudable since C(++) has never been able to do this and likely never will, but in line with the pedantry that started this whole chain of comments, it’s not memory safe.
It's hard to make predictions with an open source project, but my personal guess is some flavor of it will land (including it is already demonstrating good results without an enormous level of code complexity in the compiler and without overly slowing down compile speeds), but I guess we'll see.
It's being driven by an external contributor who has landed some good changes in the past to the Go compiler. (I think the autovectorization work might be part of their PhD or other academic research, but not sure.)
(That particular build dashboard just shows post-commit tests. '386' there is 32-bit x86, 'arm' is 32-bit ARM.)
I can't speak to what Google runs in production or what Google tests internally, but the Google-run CI for the Go project definitely does test 32-bit, and those tests essentially match the tests run against 64-bit.
I think it's trying to show a mapping operation for a generic container where the container values are of one type and the mapping function is allowed to return a container with values of a different type.
type MySlice []int
// Map maps from a slice of ints to a slice of float64s.
func (s MySlice) Map(f func(int) float64) []float64 {
var out []float64
for i := range s {
out = append(out, f(s[i]))
}
return out
}
From a quick search, this seems to be better explanation of this new 1.27 feature:
(That uses an example that seems similar in spirit to the Interactive Tour's example, but with a more useful type of a Stack[T] and corresponding explanation seem clearer.)
And to answer the second half of your request, here is that exact same code as above, but now using the 1.27 generic methods feature (with a runnable example using tip at https://go.dev/play/p/1YK62tGetsm?v=gotip):
// Map maps from a slice containing type In to a slice containing type Out.
func (s MySlice[In]) Map[Out any](f func(In) Out) []Out {
var out []Out
for i := range s {
out = append(out, f(s[i]))
}
return out
}
In short, you could always have methods on a generic type since Go first introduced generics in Go 1.18, but with 1.27, the methods on the generic type can also introduce their own additional type parameters.
(Previously, you could achieve the same net effect with a top-level generic function, but then the code would not be grouped as nicely as hanging it off of the type, and arguably it now can have slightly better ergonomics in some cases. You can see more of the rationale from Robert Griesemer at https://github.com/golang/go/issues/77273.)
> it finally took convincing Philip Wadler (of Haskell and Java generics fame) to help. Without that lucky break [...] Go probably still wouldn't have generics.
Small correction, I think:
My understanding is the core Go team was already well on its way to doing generics before Philip Wadler became involved, though he and his colleagues were later very helpful.
From Wadler's "Featherweight Go" paper [1] (the first draft of which seems to have been uploaded to arxiv.org circa May 2020):
Recently, the Go team mooted a design to extend Go with generics [Taylor and Griesemer 2019], and Rob Pike wrote Wadler to ask: Would you be interested in helping us get polymorphism right [...]
and:
It took us several tries over many months to formalise it correctly.
Well before that, doing generics was one of the main focuses in the GopherCon keynotes/announcements in 2017 and 2018 by Russ Cox [2].
The August 2018 generics design [3][4] was very detailed, and my recollection is most people in the Go community at that point believed the core Go team was going to make generics happen, even if the team and community were still iterating on the exact design.
Of course, that design improved over time, including thanks to Wadler and crew, but I think it would be overstating things to say that probably nothing would have happened without Wadler.
From what I've seen, I don't think the core Go team was ignoring the lessons of Java or C#.
Here's a sample quote from Russ Cox from 11 years ago on this site: [1]
We have spoken to a few true experts in Java generics and each of them has said roughly the same thing: be very careful, it's not as easy as it looks, and you're stuck with all the mistakes you make. As a demonstration, skim through most of http://www.angelikalanger.com/GenericsFAQ/JavaGenericsFAQ.ht... and see how long before you start to think "was this really the best way to do this?"
And of course, they asked other experts for help, including Philip Wadler (of Featherweight Java and Haskell fame): [2] [3]
We’ve been thinking about generics since work on Go began, and we wrote and rejected our first concrete design in 2010. We wrote and rejected three more designs by the end of 2013. Four abandoned experiments, but not failed experiments, We learned from them, [...]
Last year [2018] we started exploring and experimenting again, and we presented a new design [...] and we’ve been working with programming language theory experts to understand the design better.
FWIW, C# generics are way better (ergonomics) than the Java ones (due to type erasure), so Java waited too much. History is more complicated though for Java, since it was either having some generics vs having none. See [this](https://softwareengineering.stackexchange.com/questions/1766...) for a lengthier discussion.
Yes, .NET had the benefit of coming second and using F#, which was originally essentially OCaml 4 on .NET, as it's testing grounds. Same for Async
I am considering .NET for one of my compilers backends because of the reified generics. .NET can even pass around an object with generic methods that get specialized via JIT at runtime each time it sees a new data type (with reference types sharing implementations). Which also ties back to having true value types
It's a shame it took so long for .NET core to come around because even today the platform carries a reputation for being windows first which hasn't really been true for many years now
People always forget this other post when arguing for the home team.
"They are likely the two most difficult parts of any design for parametric polymorphism. In retrospect, we were biased too much by experience with C++ without concepts and Java generics. We would have been well-served to spend more time with CLU and C++ concepts earlier."
It doesn't seem particularly damning to me, though, especially written by Russ Cox in hindsight. It's almost a truism that if you had spent less time on approaches that didn't pan out and more time on approaches that did eventually pan out, you likely would have arrived at a workable solution sooner.
For example, if the core Go team had spent more time exploring C#-like approaches (as some suggest they should have), it's possible that would have delayed the whole thing.
(btw, consider me a "long time listener, first time caller" -- I pretty much always pause to read your comments while flicking through discussions here, including I appreciate you often bring in historical context, even if I might have a different take, or even if I might have expressed your take differently ;)
Go modules did not get rid of vendoring. You can do 'go mod vendor' and have been able to do so since Go modules were first introduced.
How long the google-run module cache (aka, module proxy or module mirror) at https://proxy.golang.org caches the contents of modules is I think slightly nuanced.
That page includes:
> Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package
But that page also discusses how modules might need to be removed for legal reasons or if a module does not have a known Open Source license:
> proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.
If interested, there's a good overview of how it all works in one of the older official announcement blog posts (in particular, the "Module Index", "Module Authentication", "Module Mirrors" sections there):
ok, 1) so would it be fair to modify my statement that it basically tries to cache forever unless its can't determine that its legally allowed to cache forever?
2) you're right, glanced at kubernetes (been a long time since I worked on it) and they still have a vendor directory that gets updated regularly.
> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages
Thanks for that link.
Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other languages, but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
This subthread to me was a seemingly prototypical exchange there:
Someone in that subthread tells Filippo (FiloSottile) that he is misunderstanding cargo behavior, but Filippo then reiterates which behavior he is talking about (add vs. install), Filippo does a simple test to illustrate his point, and some others seem to agree that he is correct in what he originally said.
That said, YMMV, and that overall discussion does certainly seem to have some confusion and people seemingly talking past each other (e.g., some people mixing up "dependents" vs. "dependencies", etc.).
> but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
I don't get this impression. Rather, as you say, I get the impression that people are talking past each other, a property which also extends to the author, and the overall failure to reach a mutual understanding of terms only contributes to muddying the waters all around. Here's a direct example that's still in the OP:
"The lockfile (e.g. uv.lock, package-lock.json, Cargo.lock) is a relatively recent innovation in some ecosystems, and it lists the actual versions used in the most recent build. It is not really human-readable, and is ignored by dependents, allowing the rapid spread of supply-chain attacks."
At the end there, what the author is talking about has nothing to do with lockfiles specifically, let alone when they are applied or ignored, but rather to do with the difference between minimum-version selection (which Go uses) and max-compatible-version selection.
Here's another one:
"In other ecosystems, package resolution time going down below 1s is celebrated"
This is repeating the mistaken claims that Russ Cox made years ago when he designed Go's current packaging system. Package resolution in e.g. Cargo is almost too fast to measure, even on large dependency trees.
As I understand it, part of the reason it took a while is that the core Go team was generally of the opinion that doing user-facing SIMD APIs the right way was to design a high-level, cross-platform API that would stand the test of time, and that was then punted a few times given its complexity and need to do other things.
Part of what helped the current approach take off was switching to a philosophy of designing a lower-level architecture-dependent API first (the 'simd/archsimd' package), and then later doing a higher-level portable API (the 'simd' package, which is topic of this blog post).
That two-level approach I think also gave some additional freedom for the design and implementation of the friendlier / high-level 'simd' package, including because the lower-level 'simd/archsimd' package is available for people who need or want to drop down.
It's a nice design.
reply