title: [TIL] Golang TG Community Discussion on PTT BBS Lazy Package
published: false
date: 2021-01-21 00:00:00 UTC
tags:
canonical_url: http://www.evanlin.com/golangtg-ptt/
---
# Preface
A discussion about [PTT BBS backend Golang backend partners (because it's a friend-making project, there's no salary)](https://moptt.tw/p/Soft_Job.M.1610976994.A.2C8), which also brought up discussions about PTT backend needs in the Golang TG discussion. The following is a compilation of relevant discussion content.
First, let's organize some relevant context about the partner recruitment article, with much of the content quoted from the original author.
The original PTT article is here:
- moptt https://moptt.tw/p/Soft\_Job.M.1610976994.A.2C8
- p...title: [TIL] Golang TG Community Discussion on PTT BBS Lazy Package
published: false
date: 2021-01-21 00:00:00 UTC
tags:
canonical_url: http://www.evanlin.com/golangtg-ptt/
---
# Preface
A discussion about [PTT BBS backend Golang backend partners (because it's a friend-making project, there's no salary)](https://moptt.tw/p/Soft_Job.M.1610976994.A.2C8), which also brought up discussions about PTT backend needs in the Golang TG discussion. The following is a compilation of relevant discussion content.
First, let's organize some relevant context about the partner recruitment article, with much of the content quoted from the original author.
The original PTT article is here:
- moptt https://moptt.tw/p/Soft\_Job.M.1610976994.A.2C8
- ptt.cc https://www.ptt.cc/bbs/Soft\_Job/M.1610976994.A.2C8.html
## Why is it required to read the `database/sql` and `go-sql-driver/mysql` packages in the article asking for partners?
It's mainly a matter of style. I (Pichu Chen) am very worried that someone will directly port C Code into Golang Code, and then all the comments will be written to look at which function in the C Code, then it's GG XD
## Regarding PTT BBS looking for Golang partners, what is the current goal?
At this stage, we want to first establish the necessary read and write interfaces for the overall BBS, as well as subsequent discussions and the culture of adoption.
And currently, the solution I (Pichu Chen) propose is to redesign the backend interface. We will initially get a new HTTP-based backend interface, and PTT APP middle-end or mobile APP development partners can access the BBS database through this interface. In the development process, unlike the previous BBS development process, I will first write the required functions into text documents and propose discussions, and then open a GitHub ISSUE for implementation after a period of time. Therefore, it can ensure that the new code has documentation and clear and easy-to-understand test cases, avoiding repeating the same mistakes. Currently, we have completed functions such as verifying accounts, obtaining board lists, obtaining article lists, and obtaining article content. We also need to continue to complete functions such as adding push/recommend, adding articles, and editing favorites. But I have an additional request. Because of the experience of the "Tokyo Metropolitan Government's COVID-19 Countermeasures Website (https://stopcovid19.metro.tokyo.lg.jp/)" mentioned earlier on Soft\_Job, I still hope that this project can be completed by the majority of the community, rather than, like most open-source projects in Taiwan, being completed by a fixed number of "gods."
## Current problems with PTT (arranged by urgency)
1. The code for the interface/business logic/database is mixed together, making it difficult to adjust the user experience and user interface.
2. The code lacks comments and has extremely low readability.
3. The original code has no testing code at all.
4. The code has no benchmark mechanism at all, and modifying the architecture relies on the authority of the designer rather than scientific evidence.
5. Most of the architecture still uses a 32-bit time representation, which will lead to the 2038 problem.
6. Passwords still use DES-based hashing, in other words, the strength is insufficient.
7. Excessive reliance on shared memory design makes server distribution difficult.
8. The index file storage method is not flexible enough and is not easy to add new fields.
9. The mail forwarding mechanism has been dead for a long time.
10. In-station messages (water balls), in-station letters cannot notify users in real-time via mobile phones.
11. Current PTT code does not support IPv6.
12. In-station articles still use Big-5 storage and do not support emoji or Taiwanese Romanization.
13. Does not support image uploads, audio, or video communication.
## So, reading SQL packages is not about writing PTT data into a database?
Actually, BBS articles are quite hierarchical, that is, the article itself has a unique id, and then the board ref a series of ids, which PTT already has, which is their own URL conversion system (like: https://www.ptt.cc/bbs/car/M.1611180581.A.E43.html) This kind of M.161180581.A.E43, so since there are ready-made keys and key generators, this aspect should not need to be redesigned. In addition, pictures and log...er... the data of bbs should mostly not be pictures or logs.
## If PTT data is written into the database
Then the discussion history of whether to convert the BBS backend database to SQL has been quite long, but in conclusion, after several years of discussion, there is really no one who has completely become a full SQL architecture (maybe Bahamut has? After all, they are not open source, so I don't know). But at this stage, I want to first establish the necessary read and write interfaces for the overall BBS, as well as subsequent discussions and the culture of adoption, because there were indeed many discussions about whether to convert to SQL before, but the actual performance tests were almost non-existent. After the interface is established, try another branch that is implemented with SQL. After there is performance data, this problem will be easier to reach a conclusion in the discussion.
## If PTT data has the opportunity to be put into RMDB
If it is to convert the original database to a SQL series of relational databases or MongoDB, I think the advantages will be that these are database tools familiar to modern developers, and then the performance part has other teams to help with the quality control, and there are ready-made distributed database designs.
The disadvantages I asked senior students several years ago, the larger part is that SQL Command needs to be parsed separately, which will have additional costs, and then not everything is suitable for blindly stuffing SQL, such as pictures or Log file design, I may not put it on a relational database, I guess AWS S3 won't either.
But it's not to say that SQL is useless in the BBS project. I remember that in PTT BBS, they put the push text in a relational database, but I'm not sure if it's the general PgSQL, MySQL or Sqlite (I guess it's sqlite, because I saw similar code)
As for whether the modern new design should or should not use FileSystem as DB, I think it still depends on the problem itself. For new projects like CoreDNS, he does save the issued lease in a space-separated format. He can actually use sqlite, but I think if he uses sqlite, he needs to introduce an external package. Then, if the online OP needs to directly delete a certain lease record, he needs to issue a DELETE command, which is not faster than opening vim and deleting it.
Furthermore, the advantages of modern relational databases, in addition to providing a unified operation interface, are that it has a b+-tree-based index, which is not seen in the current BBS system. Currently, the accelerated index seen in the BBS system will be like I want to find pichu and then classify by the first letter p, and then take the things under the /usr/p/pichu folder. As we all know, the frequency of e and i is usually higher, so it is very likely that the number of users or boards in this folder will be more than other folders, causing a slight imbalance. A better approach might be like the git database design, first hash the userid with sha1, and then also take one or two digits to make a folder index, which is easier to balance.
But the existing article-based approach does have some shortcomings. For example, the minimum file size will be limited by the file system's limitations. If the block size of a certain file system is 4KB, then even if it only has a line of 12 Byte text, it will also occupy 4KB and an inode in the hard disk, and therefore, each station will later promote the prohibition of "one-line articles", and even later developed a push text system, which is basically to deal with this problem. But modern hard disk space is actually much larger, and the number of inodes is also much larger, whether it is still a problem can also be re-explored. Or add some modern mechanisms to count how many times articles have been modified for more than a year, which are unpopular articles, and then predict that the probability of being modified in the next year is less than a p value, and then archive them into the same large file, and use another sqlite file to index their fseek position and length (if you want to be more open-minded, you can also add compression to save space.
That is to say, when the predecessors designed this system, it was not like "using a database", but rather to discuss how to design a database, and then different BBSs would refer to each other, and then it became what it is now.
As for these discussion articles, you actually have to look through the various small stations. It should be difficult to find them on Google, or like this article, when the time comes, it will be washed to who knows where by telegram~
# Others:
- PTT BBS source code https://github.com/ptt/pttbbs
- PTT File struct https://github.com/ptt/pttbbs/wiki/STRUCTURE
- MapleBBS 3.0 presentation document https://hackmd.io/@holishing/BkJHevM4f?type=view
# TG Discussion Full Text:
For query purposes
hsu jimmy, [20.01.21 19:04] [Recruitment] BBS backend implementation (all remote) (unpaid) https://moptt.tw/p/Soft_Job.M.1610976994.A.2C8
Wu Gordon, [20.01.21 19:06] [Photo]
Wu Gordon, [20.01.21 19:06] I’m curious why it’s needed
Stefan ﾠ😹, [20.01.21 19:07] [In reply to Wu Gordon] Implies that ORM is not used
Wu Gordon, [20.01.21 19:08] But there is no need to modify it to your own version, why is it necessary to have read it? The package doc usage specification should be able to satisfy the usage 🤔
Wu Gordon, [20.01.21 19:08] I’m just curious about the thinking behind this requirement
Stefan ﾠ😹, [20.01.21 19:09] That is to say, SQL is heavily used
Stefan ﾠ😹, [20.01.21 19:09] No form of packaging is done well
Stefan ﾠ😹, [20.01.21 19:09] Like sourcemod’s messy sql
Evan Lin, [20.01.21 19:13] [In reply to Stefan ﾠ😹] In addition to this, I may want to understand the writing style of these two packages. This can also ensure that you know that the other party’s writing style can be based on the style of these two packages?
Wu Gordon, [20.01.21 19:14] So that’s it
Julian Chu, [21.01.21 07:34] It looks more like borrowing the concept and architecture of database/sql to provide a common interface, and swapping the driver can access BBS file data in different formats
Rayer Tung, [21.01.21 07:47] Speaking of which, isn’t PTT using a general SQL (seems to be Postgres)?
Rayer Tung, [21.01.21 07:48] What reason is there to do an adapter yourself instead of using the existing implementation?
Rayer Tung, [21.01.21 07:49] Each implementation actually uses a common interface
Yami Odymel https://yami.io/, [21.01.21 07:54] A mystery of the century
Julian Chu, [21.01.21 08:27] I don’t remember PTT using SQL, where can I see this information?
Peiming, [21.01.21 08:27] [In reply to Rayer Tung] Because no SQL is used, one article is one file. https://github.com/ptt/pttbbs/wiki/STRUCTURE
Julian Chu, [21.01.21 08:28] [🍺 Sticker]
Peiming, [21.01.21 08:28] More information needs to be found
Rayer Tung, [21.01.21 09:40] ah, make sense.... I always thought he had imported it into sql before. But in this case, the first thing should be to import this system that uses FS as DB into a real DB o_o
Rayer Tung, [21.01.21 09:41] Instead of trying to hardcode an FS as DB Adapter
Rayer Tung, [21.01.21 09:41] [In reply to Julian Chu] This is in my memory, but obviously I remembered it wrong 😆
Peiming, [21.01.21 09:45] [In reply to Rayer Tung] This idea is mainly to allow each BBS to write its own driver, because the BBS used by each station is not the same now https://github.com/clyang/bbslist
Rayer Tung, [21.01.21 09:47] This plan is quite grand 😆 But I think that, assuming I can make the decision, I will first convert PTT to DB, and the priority of this "adapter for other bbs" I will put at the very end. But maybe they have more reasons that I don’t know, maybe.
Rayer Tung, [21.01.21 09:47] After all, using FS as DB made sense 25 years ago, but it is very incomprehensible now (or even ten years ago).
Pichu Chen, [21.01.21 14:57] [In reply to Rayer Tung] Hello,
If it is to convert the original database to a SQL series of relational databases or MongoDB, I think the advantages will be that these are database tools familiar to modern developers, and then the performance part has other teams to help with the quality control, and there are ready-made distributed database designs.
The disadvantages I asked senior students several years ago, the larger part is that SQL Command needs to be parsed separately, which will have additional costs, and then not everything is suitable for blindly stuffing SQL, such as pictures or Log file design, I may not put it on a relational database, I guess AWS S3 won’t either.
But it’s not to say that SQL is useless in the BBS project. I remember that in PTT BBS, they put the push text in a relational database, but I’m not sure if it’s the general PgSQL, MySQL or Sqlite (I guess it’s sqlite, because I saw similar code)
As for whether the modern new design should or should not use FileSystem as DB, I think it still depends on the problem itself. For new projects like CoreDNS, he does save the issued lease in a space-separated format. He can actually use sqlite, but I think if he uses sqlite, he needs to introduce an external package. Then, if the online OP needs to directly delete a certain lease record, he needs to issue a DELETE command, which is not faster than opening vim and deleting it.
Furthermore, the advantages of modern relational databases, in addition to providing a unified operation interface, are that it has a b+-tree-based index, which is not seen in the current BBS system. Currently, the accelerated index seen in the BBS system will be like I want to find pichu and then classify by the first letter p, and then take the things under the /usr/p/pichu folder. As we all know, the frequency of e and i is usually higher, so it is very likely that the number of users or boards in this folder will be more than other folders, causing a slight imbalance. A better approach might be like the git database design, first hash the userid with sha1, and then also take one or two digits to make a folder index, which is easier to balance.
But the existing article-based approach does have some shortcomings. For example, the minimum file size will be limited by the file system’s limitations. If the block size of a certain file system is 4KB, then even if it only has a line of 12 Byte text, it will also occupy 4KB and an inode in the hard disk, and therefore, each station will later promote the prohibition of "one-line articles", and even later developed a push text system, which is basically to deal with this problem. But modern hard disk space is actually much larger, and the number of inodes is also much larger, whether it is still a problem can also be re-explored. Or add some modern mechanisms to count how many times articles have been modified for more than a year, which are unpopular articles, and then predict that the probability of being modified in the next year is less than a p value, and then archive them into the same large file, and use another sqlite file to index their fseek position and length (if you want to be more open-minded, you can also add compression to save space.
That is to say, when the predecessors designed this system, it was not like "using a database", but rather to discuss how to design a database, and then different BBSs would refer to each other, and then it became what it is now.
As for these discussion articles, you actually have to look through the various small stations. It should be difficult to find them on Google, or like this article, when the time comes, it will be washed to who knows where by telegram~
Liu Kakashi, [21.01.21 15:01] It’s written very well, it should be preserved
Rayer Tung, [21.01.21 15:04] Actually, BBS articles are quite hierarchical, that is, the article itself has a unique id, and then the board ref a series of ids, which PTT already has, which is their own URL conversion system (like: https://www.ptt.cc/bbs/car/M.1611180581.A.E43.html) This kind of M.161180581.A.E43, so since there are ready-made keys and key generators, this aspect should not need to be redesigned. In addition, pictures and log...er... the data of bbs should mostly not be pictures or logs. I also know a little bit about the concerns of the MAPLE era. SQL may not run as well as FS back then. But modern DBs have indexes, redundancy, and partitions, all of which can greatly increase read and write efficiency. Of course, MAPLE is already over 20 years old, and it is indeed very difficult to change this. But I think writing a go sql lib compatible interface compared to this, which investment is good and which is not good, I think it can be discussed? 😆
Rayer Tung, [21.01.21 15:05] Especially FS backup, I guess they use Sunday morning rsync? If so, DB should do much better....
Rayer Tung, [21.01.21 15:05] But whether PTT still has such a high investment value, I respect the opinion of the person in charge, after all, in consensus, this is a sunsetting thing, but we can see if we can make him go a little longer
Pichu Chen, [21.01.21 15:33] [In reply to Rayer Tung] Basically, there is no need to be compatible with the sql lib. The reason why I asked everyone to look at these two projects at that time was also mentioned by someone in the TG group above. It’s mainly a matter of style. I’m basically very worried that someone will directly port C Code into Golang Code, and then all the comments will be written to look at which function in the C Code, then it’s GG XD
And the discussion history of whether to convert the BBS backend database to SQL has been quite long, but in conclusion, after several years of discussion, there is really no one who has completely become a full SQL architecture (maybe Bahamut has? After all, they are not open source, so I don’t know). But at this stage, I want to first establish the necessary read and write interfaces for the overall BBS, as well as subsequent discussions and the culture of adoption, because there were indeed many discussions about whether to convert to SQL before, but the actual performance tests were almost non-existent. After the interface is established, try another branch that is implemented with SQL. After there is performance data, this problem will be easier to reach a conclusion in the discussion.
Rayer Tung, [21.01.21 15:35] I just think that creating a sql interface may be quite difficult, and what’s more worrying is whether the performance of each method will be sacrificed because of FS read and write. But indeed, you have to do this to know, it’s not very meaningful to guess without anything to benchmark, I think there is no problem in this direction. Leading this year-old accumulated stuff is actually quite complicated, come on.
Rayer Tung, [21.01.21 15:36] Since it’s just for style unification, I think there will be a benchmarkable version soon
Evan Lin, [21.01.21 16:04] [In reply to Rayer Tung] Maybe it depends on what the overall operational idea is, after all, ptt -> db is equivalent to rebuilding the architecture. Regardless of legacy concerns, it may be more important for other bbs adapters. It was really tiring to connect in the Maple era~ You have to run another program to slowly align (twbbs?)
If PTT can be used as a single source of truth, it can provide many APIs for more people to brainstorm and use. It’s great~ APIs are out~ I think more applications will be brainstormed (I’m looking forward to it)
Kevin Yang, [21.01.21 16:06] PTT may be gone soon
Kevin Yang, [21.01.21 16:07] Not open for registration, popularity has been declining
Kevin Yang, [21.01.21 16:07] And all kinds of police
Rayer Tung, [21.01.21 16:07] [In reply to Evan Lin] Actually, making the fs based structure of PTT Restful is equivalent to making the MAPLE series of bbs Restful. I’m looking forward to that day 😆
Marcus Liu, [21.01.21 16:44] Actually, PTT has been calling for help They also want to solve the registration problem first
Marcus Liu, [21.01.21 16:44] https://g0v.hackpad.tw/Ptt–ctwZwU7BxcJ
Julian Chu, [21.01.21 16:44] Even if it’s converted to sql, the parser for the existing data still has to be done, the risk of directly converting to sql is higher, providing an interface to increase flexibility, and being compatible with the existing data format first, I personally think it’s a low-risk choice that can see results faster
Marcus Liu, [21.01.21 16:44] But to be honest, I think it’s very difficult
Evan Lin, [21.01.21 16:54] [In reply to Pichu Chen] @pichuchen and @killercat9 If you don’t mind, I’ll organize the content of this discussion into a blog and then repost it to the Facebook group. It won’t be buried in the torrent of time
Pichu Chen, [21.01.21 16:54] I’m OK CC BY-SA
Rayer Tung, [21.01.21 16:54] No problem
Rayer Tung, [21.01.21 16:55] CC BY SA