Scraping threads from Corona forum to PDF for Google NotebookLM

[mase]

So recently Google has posted a research tool called NotebookLM, and to put it simply it's a custom AI bot (similar to Chat GPT) that only looks up and answers questions found in data one provides. Unlike traditional AI bots, it doesn't hallucinate information if it doesn't know it and unlike other bots can answer you 'I don't know' which can be very useful.

People started to use it to make custom AI bots that are very niche and topic-specific which basically act as a big manual for your chosen thing, except that it has AI that's quite efficient at this. So my thought was - would it be possible to download certain threads of this forum, put it into a PDF and use it as a source? There's a lot of valuable information here (and a lot of not so valuable, opinions, out-dated information etc. but still) and it can get troublesome when sometimes you need to find a specific topic or question.

I tried making a quick python script that fetches information from threads but it's not as easy as I thought and perhaps there's an easier solution that I don't know. I'm also curious if anyone else is using the tool or finds this idea interesting and perhaps relevant for them.

[maru]

We already have a chatbot at https://support.chaos.com/hc/en-us which is trained on our Help Center and Documentation.

[mase]

I see. What about other information on this forum? People discuss in all kinds of topics related to rendering, hardware, workflows etc. I, for example, have a few threads bookmarked for when I need to come back to certain topics and look up information because I know it's credible and it works. We could basically have a forum chatbot, and of course as I mentioned previously not all of the information here is credible or still valid but that's for everyone to decide I guess.

[maru]

Some updates to the forum are planned, so we might consider a "forum chatbot" or a similar solution as well.

[mase]

That would be nice. Not sure if NotebookLM allows to integrate its' chatbot elsewhere but I would find this feature very valuable (and I believe some others too).

We could also avoid having duplicate threads for the same topic (because forum search, or even google search for that matter doesn't always find other threads on the same topic). Another very useful feature that Notebook LM chatbot has is it references sources after each statement so one can go and check it themselves. I imagine this would also help when looking for a particular thread or post and not remembering the exact title or content of the message (I was looking for an old Juraj's project post sometime ago and even fully knowing what it looks like and vaguely remembering the title could barely find it).

Besides that, I'd say forum works pretty well as it is so I'm interested to hear what are the main reasons to update it (if not for an easier/better search).

[TomG]

Lots of reasons, the better search is a primary one because the search here is terrible :) But also improved look (as this one looks very old), easier way to work with multiple products without having different forums to have to visit, generally improved performance and functions/options (again, this one is very old and out of date), better ability for individuals to customize things (what they see, what they see first, etc.), better protection against bots and spam attacks, and probably more :)

[pokoy]

All the obvious (and questionable) use cases aside - have users ever agreed to their posts (and potential solutions and work results) being scraped off the forum like that? Don't think so... and honestly, wtf?
Posting to a forum and having it visible to forum members is different from having someone scraping it for 'content gain'. Do the leg work and learn on your own just like everybody else.

[mase]

All the obvious (and questionable) use cases aside - have users ever agreed to their posts (and potential solutions and work results) being scraped off the forum like that? Don't think so... and honestly, wtf?
Posting to a forum and having it visible to forum members is different from having someone scraping it for 'content gain'. Do the leg work and learn on your own just like everybody else.

Well, in a way there's no one stopping you from just going through each forum thread, right-clicking and pressing Save Page As... I believe by posting something on this forum you also agree to have that information public. Perhaps using this information to then gain something monetary out of it or distributing in some way could be questionable but other than that I don't see how archiving public information crosses any ethical boundaries (if that's what you're trying to imply).

Essentially, I see this Google chatbot as a more advanced search engine that is also able to connect various findings between the sources it's been provided and summarize it. Similar tools have been on the market forever.

[pokoy]

I might have agreed to post on a forum and have others see my posts - that's about it, not more than that.
So, sure, go on and have a blast. Just make sure my posts aren't included in your 'research'.

[mase]

I might have agreed to post on a forum and have others see my posts - that's about it, not more than that.
So, sure, go on and have a blast. Just make sure my posts aren't included in your 'research'.

That's not really my goal here, and if I'm in wrong here with this concept then I won't proceed with the idea. Besides, I don't even know how one would implement it and was more interested in discussing a tool that would be useful not only for me but for all the community. If it's not within forum rules then I guess there's no reason to continue this discussion.

[pokoy]

I might have agreed to post on a forum and have others see my posts - that's about it, not more than that.
So, sure, go on and have a blast. Just make sure my posts aren't included in your 'research'.

That's not really my goal here, and if I'm in wrong here with this concept then I won't proceed with the idea. Besides, I don't even know how one would implement it and was more interested in discussing a tool that would be useful not only for me but for all the community. If it's not within forum rules then I guess there's no reason to continue this discussion.

I see. I'll try to distant myself from my negativity towards AI for the sake of constructive feedback... Let's say you were able to achieve what you want - what exactly would it help you with? Tell you what settings to use in a certain scenario? Tell you why your lighting doesn't look good or the camera renders black? There are so many different variables in almost any 3d related job, it would be impossible for an AI to correctly understand your problem and suggest a possible solution. After all, I don't think it would actually be able to be as concise as another experienced user or support staff. There's an extensive online reference plus this forum with real people who can help with almost anything you could come up with, so what exactly is missing right now that you're trying to solve?

Don't get me wrong, but your request sounds like the common 'let's throw AI at something and expect magic to happen with rainbows on the side' techbro wet dream. It can only disappoint.

FWIW, I think that the company running this place should prohibit anyone from scraping the content (or to clearly communicate it's not allowed, even if it can be worked around by simply creating a profile). After all, some people here have chosen to share private data, the staff should at least realize there's a risk of abuse of personal data by using automated tools. Just to shrug it away casually is not OK in my opinion and I'm surprised no one cares.