Does it matter whether the data was poorly secured? LLMs should not be hacking into government medical websites, and if they do, the companies responsible should disclose the incidents as soon as possible.
The problem is that sufficiently poor security is indistinguishable from authorized public access. And unfortunately a lot of real world "digital security" is in fact that bad.
A lot of these "hacks" are the equivalent of asking "hey, can I come in?" and the guard assuming that anyone who would ask is authorized, and thus saying "yes". But if the guard said "yes" then it seems a bit absurd to call it trespassing.
lets say you have page=0 some of these pages are public and some are private, and the only way you secure the private pages is to not link it on the website.
is incrementing a url query parameters from 0 -> 1 count as hacking?
Is it robbery when in this case it was a sign with information that you were planning on putting on your front fence for public display but while preparing it was left sitting in the front yard with a small fence.
If I walked past, saw it and remembered it or even recorded it, is that honestly theft?
Yep, because private enterprises never go bankrupt, commit fraud, mistreat customers, bribe politicians, put profit ahead of safety, hack websites, etc. etc.
I would take "Our town has the castle" to imply that there's a group of towns that collectively only have one castle, and your town is the one that has it.
Compare: "Sacramento is the capital." You could say Sacramento is a capital if the reference group includes Albany, Austin, Tallahassee, etc. Or you could say Sacramento is the capital if the reference group includes Los Angeles, San Diego, San Francisco, etc.
There's an interesting prompting technique for this that asks the model to start with a random string that is then manipulated into the answer: https://pub.sakana.ai/ssot/
All instructions to LLMs are merely suggestions to nudge it in the right behaviour. Unless you have a deterministic guardrail that guards against a single specific action, everything else is a slot machine that's biased strongly in your favour.
My global CLAUDE.md explicitly states "When commenting on code and configs, or writing MD files, strictly write within the domain of the content being commented on. DO NOT include information, negatives or ramblings from work sessions. For e.g. if commenting on a proto string field that is replacing an int field, do not comment that 'this is not an int field'".
This reduced the idiocy of the agent (Opus 5 included) when writing documents. But I'm still catching it writing README.md talking about the negatives that it removed. Those belong in the memory if it is actually that important (most of the time it's junk), but Claude doesn't seem to understand and never ever learns.
> But there is one huge question: did Tristan opt out of model training for his ChatGPT and Codex sessions? If the answer is no, then this seems fair game.
Just because something is legal and permitted by terms of service doesn't mean it's morally right.
>Just because something is legal and permitted by terms of service doesn't mean it's morally right.
What are you expecting OpenAI to do exactly if these mathematicians voluntarily submitted their prompts into ChatGPT's training data? Are they supposed to manually review all their data to make sure competing mathematicians didn't accidentally leave the "submit prompts" toggle on?
Or were they supposed to not try to solve Navier-Stokes, or were they supposed to just not tell anyone that they had solved it?
To me it’s morally ambiguous… if you hand parts of your thinking over to a tool like this (knowing full well the terms of service), of course the tool makers will want to claim some credit, and they do deserve it. But the bigger question to me is the scientific one: did their new model arrive at this result because it had closely-related training data from a human, or did it extrapolate to this line of thought on its own? The answer says a lot about how valid their claims of “AGI” are vs. a very fortuitously cherry-picked example.
It would actually be a really interesting study, if they would ever be willing to be transparent about this, how the result differs with and without his conversations in the training set. How quickly it arrives at the result, whether it takes the same approach, etc.
> What are you expecting OpenAI to do exactly if these mathematicians voluntarily submitted their prompts into ChatGPT's training data?
Personally, I would expect them to have a little class, to KYC, and to manually turn off training for known competitors using their service so as to avoid any unforced goofs like this.
> Are they supposed to manually review all their data t
Yes. They should determine if training data included this teams data. Consider the money they spent, the press release and the purpose of their publication.
Since they failed to answer this question they shouldn't have published.
- Good design requires user interviews, which are less common in open source
- The sorts of people who are most attracted to open source work are programmers, not designers
- Open source software can get popular even with bad design; closed source software with bad design fails and shuts down
Designers have it really hard in free software because good design is about understanding the big picture, knowing what to leave out, have product vision to follow.
That's tough to do in environment where goal is to make everyone happy and the biggest say have programmers who are developing the thing who are usually not interested in design, inexperienced in design or have their personal very specific product vision.
I think you definitely identified the issue well. Changing the design of an existing project, especially if you're not already involved in the project is a daunting task. You have to try and convince a core group of users that the design is going to be better to entice new users. But the problem is that those core group of developers likely don't care that the UI isn't up to par, and likely don't want it to change.
I even remember having this problem personally when trying to improve designs of webpages at my workplace. We had many of our technical users still using the command line for their mail, and even for browsing some webpages. Their whole workflow was stuck in the command line and they've been doing it that way for 20+ years. Trying to change things to improve the experience for others was like pulling teeth. They would retort the classic "Why change it? It works the way it is, no need to change it". Which to a degree I can understand. Some websites out there really do change their UI's for completely silly reasons and sometimes even lose functionality because of it.
At least this is one positive thing coming from AI driven development. People can actually make some pretty decent UI's without having to know as much about good UI design. AI can try to utilize the "best practice" standards and make a UI that often looks much better than most devs could make. There are some downsides of course, such as the fact that almost every new website nowadays looks the same, since Claude seems to have one style it always goes with. But maybe that tradeoff is okay.
I don't agree about the AI at all. AI is terrible at design (compared to programming).
Having relatively nice UI skin doesn't equal good design. They are surprisingly unrelated as i am sure people know highly usable well designed software with bad UI and also know super beautiful UIs that are unusable - badly designed.
AIs are bad at design because they don't understand and can't keep context of the whole together. Just like in programming they are much better at solving single well defined problem. Design is very much opposite of that, it's always about reharmonizing in context with the whole after you make a change.
They have the SEO juice to push out and replace dedicated wikis as the top result, even when there are long-standing dedicated wikis for the game/universe. Fandom wikis then typically have maybe half of the information from the dedicated wiki PLUS a bunch of intrusive ads. There's no instance that I know of where a Fandom wiki popping up for a community was a positive unless they simply didn't have one to start with.
Well, I'm an American, but US media is disproportionately set in Los Angeles or NYC, and I haven't been to either of those places.
I was watching Monk recently, and it feels kinda neat seeing a show shot in San Francisco because I used to live there. I wouldn't say it feels "cheap" though.
reply