Bug 271826

Comment 1 Warner Losh 2023-06-04 22:23:46 UTC

Is a -current snapshot any better? Does 12.x work any better?

Comment 2 Graham Perrin 2023-06-05 00:47:51 UTC

(In reply to Sergey Fedorov from comment #0)

How are the HDD(s) and SSD(s) connected?

> … when installer boots, sometimes there are numerous USB-related errors, 
> but sometimes they are skipped. …

In the absence of a photograph, I might assume hardware issues.

Created attachment 242607 [details]
An instance of USB errors

(In reply to Graham Perrin from comment #2)

I do not think so, since it happens on two different machines which otherwise work perfectly fine.
However I add a pic now of USB-related issues.

Please notice those are not central to the matter though. Freezing happens regardless.

Created attachment 242608 [details]
USB error upon intro screen load

This happened 2–3 times out of numerous install attempts. I.e. not consistent, nevertheless did happen on both machines.

I will re-upload pics, compression killed them. Sorry.

Created attachment 242609 [details]
USB errors when booting (resolution fixed)

Created attachment 242610 [details]
USB error upon intro screen load

Created attachment 242611 [details]
Booted system on PMG5 2.3

Installed and booted system on PowerMac G5 2.3 DC. No errors on boot, but it was badly freezing even when entering login/password and every other command.

(In reply to Graham Perrin from comment #2)

> How are the HDD(s) and SSD(s) connected?

Native SATA bus connection. I.e. not a PCIe card, FW drive or anything else non-default.
SSD were a direct replacement for HDDs. The PowerMac where I had FreeBSD installed now has HDD in upper bay which is used for FreeBSD (I moved macOS onto SSD).

(In reply to Warner Losh from comment #1)

I don’t have spare DVDs around to check that. Is there a way to make install image recognizable from a FW drive? That works fine for MacOS installation, but I could not get it work for FreeBSD. USB drive might be an option too, late PowerMacs support USB booting, unofficially.

In attempting to install FreeBSD on PPC I found that utilizing the 13.2 DVD iso with a G5 1.8 Ghz single processor and Nvidia FX5200, the install was smooth and rapid, working nicely in text mode (but no success on the graphic side). Moving forward I attempted an install of 13.2 on my favored device, a G5 1.8 Ghz dual processor with an ATI 9600XT card, but the install was painfully slow... marginal graphic support (card is "supported"). So not to be thwarted, I pulled out my last remaining G5, an identical G5 dual 1.8 with the 9600 card and attempted the same install... once again, the process was incredibly slow with the same outcome observed as with the first dual. Just to confirm my suspicion, I wiped the drive on the original G5 single processor and began an install. Once again, with the single, the process was quick and smooth.  So basically I found 13.2 very workable on a G5 single.  However, it has been my experience and that of others that install on a multiprocessor PPC is exceedingly challenging and as noted by opening poster renders 13.2 essentially unusable on these devices.
Good news:  Swapping in a 9600Xt card into the G5 single has yielded fully functional FreeBSD on PPC.
Bad news:  13.2 would currently seem to be a no go on the multiprocessor devices.
Hopeful that this may addressed and remedied to some extent.
P.S.  all drives standard HDD.

Comment 13 Justin Hibbits 2023-06-12 15:06:21 UTC

Does 13.1 work on multiprocessor/multicore G5 machines?  Unfortunately my FreeBSD G5 died 5 years ago, so I haven't been able to maintain the G5 at all, I've been using only the Power9.

I recommend testing 12.x (latest), 13.1, and 13.0 (if 13.1 is slow).  That could help narrow down when the slowdown occurred.

(In reply to Justin Hibbits from comment #13)
To answer your question I've tried several release on my PowerMac G5 Quad (PowerMac 11,2 with pcie bus) using the DVD drive.

Here are the results

12.4 Boots and installs just fine.

13.0 Kernel loads but painfully slow, very similar behaviour to Sergey's issue.
Loads normally until "Root mount waiting for: usbus2" then fans are roaring and takes forever to get to the Install menu.

13.1 Pretty much the same as 13.0, except even worse as it simply reboots on its own from time to time.

13.2 Doesn't load at all, Black screen with fans roaring immediately.

14.0 Latest Snapshot, Same observation as 13.2.

According to what I've seen, the Toolchain/ABI starting with release 13 has departed from GCC4 which makes it even more red herring.

It would be really awesome to get a PPC970/MP functional 64bit release with modern toolchain ! 

I'd like to help testing/troubleshooting as I fear this is the latest opportunity we have until this platform fall into oblivion. 

Cheers,

Michael

(In reply to Mitch from comment #14)

Thank you very much for testing!

P. S. It would be great if someone could address this and fix in the current master of FreeBSD.

There are many people using PowerPCs, and it remains the only affordable/accessible option for the most end-users as well as developers. Given that other realizations of BSD apparently are fine, it is certainly not something unfixable.

Any update on the matter from PowerPC port maintainers?

We have FreeBSD working on G4 but broken on G5, this is very upsetting.

FreeBSD 14 (ppc 32-bit) does not boot on G5 at all. Tries to load in OF, then quits. (I have probably not tried 13.x ppc on G5, not sure if that one works. Initial report was about ppc64 version.)

I can confirm this behaviour. I had a fine running FreeBSD 12 test disk which I nulled to try the latest stuff (14). But the 14 installer (RELEASE and STABLE from 18th of April) is so slow that it is impossible to use.

can also confirm now that 12.4 is installing and running fine on a dual g5 powermac.

Comment 20 Justin Hibbits 2024-05-01 13:57:57 UTC

(In reply to Sergey Fedorov from comment #15)

Unfortunately I think the number of committers with PowerMac G5 hardware supporting FreeBSD is dwindling as the machines age.  My only G5 that ran FreeBSD died back in 2018, so I'm unable to do anything significant for it.

That said, I'll support as best as I can.

Regarding the "USB error upon intro screen load" that's often solved by unplugging/plugging the device (happens all the time with one of my FTDI debuggers), not a problem with FreeBSD itself.

The "Root mount waiting for usbus2", unless you have root on USB you can try setting the hw.usb.no_boot_wait tunable; it looks like usbus2 is having enumeration problems (just a guess).

For 13.x performance, you can try testing a commit before, and after, c583b02587 (PowerMac timebase sync for G4), which should be a nop for G5, but might not be.  Other possibilities are pmap related (superpage support was added in the 13 timeframe, I believe).

If anyone can build a kernel without c583b02587 to test I can try that. Since 12.4 has not even working packages anymore I can't do anything.

(In reply to Denis Ahrens from comment #21)

I can try to cross compile a 14 PPC64 Kernel without c583b02587 on my x86_64 server if I can find instructions on how to achieve this.

I also have a functional PowerMac 11,2 Quad to test.

Comment 23 Warner Losh 2024-05-27 20:42:41 UTC

(In reply to Mitch from comment #22)

You can do something like:

git checkout main
git checkout -b tracking-branch
git revert $HASH
make kernel-toolchain TARGET_ARCH=powerpc64
make buildkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64

will build it. you can install it to the destdir fo your choice:

mkdir $HOME/powerpc-test-kernel
make installkernel KERNCONF=GENERIC64 TARGET_ARCH=powerpc64 NO_ROOT=t

(In reply to Justin Hibbits from comment #20)

If someone could build the image to try (or if installing from a FireWire drive works, then I can clone DVD image there and just replace the kernel, but I still need the new kernel), I will definitely test it.

Hello everyone,

In my attempt to build a Kernel without c583b02587, I was initially confronted to this conflict below 

mitch@bsd:~/freebsd-src $ git status
On branch tracking-branch
You are currently reverting commit c583b0258728.
  (fix conflicts and run "git revert --continue")
  (use "git revert --skip" to skip this patch)
  (use "git revert --abort" to cancel the revert operation)

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   sys/conf/files.powerpc
	modified:   sys/powerpc/powermac/macio.c
	modified:   sys/powerpc/powermac/platform_powermac.c

Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by them: sys/powerpc/powermac/platform_powermac.h
	deleted by them: sys/powerpc/powermac/tbgpio.c


I've simply removed both files mentioned above in the unmerged section and I was able to build a Kernel which I assume was the right thing to do, if not please let me know.

Thanks to Warner's instructions I then cross compiled a test kernel against the main branch from the freebsd source tree which appears to be the current 15 revision.

You can find it here :

https://box.ochem.net/cloud/index.php/s/Q6aCBiFdWrk69rb

I haven't test it myself but I assume you could simply rename it to kernel.txz and do an in place replacement for testing purposes.

If needed I could also build a Stable/14 version of it.

Cheers,

Mitch

I tested the kernel from Mitch and sad news is that the problem is still there.

Comment 27 Justin Hibbits 2024-05-30 18:31:10 UTC

(In reply to Denis Ahrens from comment #26)
Other big changes that are PPC970 relevant are in the pmap logic.  If you take a look at the git log of sys/powerpc/aim/mmu_oea64.c and sys/powerpc/aim/moea64_native.c, you can try to revert (git revert, or a spiritual revert if there are conflicts) some of them and see what impacts.

Some options:

e2d6c417 (superpages)
a79540111 (micro-optimizations of PVO-PTE logic)
bc94b7009 (re-merge ISA3 HPT with moea64 native HPT) -- this one *will* have conflicts.

Another shot in the dark is e44ed9d34 (atomics)

This is all just a guess based on trimming through `git diff origin/releng/12.4..origin/releng/13.1 sys/powerpc`.  There may be other issues, but these are the first I see as possibilities.

Created attachment 251098 [details]
timebase sync hack

I'm not sure my symptoms exactly match what's described here, but it seems like timebase sync may be broken on G5.  ofwcons freezes for about a minute every so often, the fans go full blast at the same time because powermac_thermal is hung.  The system is usually (always?) responsive over ssh while this is happening.

Anyway, the attached hack seems to help in my case, as least.  Only tested on the later PCIe dual and quad G5s.  I'm not sure timebase sync is even supported on PCI-X powermacs (definitely not on my Xserve G5 with weird sync via I2C).

Comment 29 Justin Hibbits 2024-05-31 20:56:55 UTC

(In reply to ben from comment #28)
I think your patch is in the right direction, but I think the explicit memory barriers are overkill in part.  We likely need to use acq/rel semantics with the atomics, and always do a load_acq() when checking cpu_done.

Something like using:

atomic_add_rel_int(&cpu_done, 1); (in both cases)

while (atomic_load_acq_int(&cpu_done) < mp_ncpus)
    ;

And maybe do the same for tb_ready.

Created attachment 251202 [details]
tbsync patch 1

(In reply to Justin Hibbits from comment #29)
Okay.  Thanks for the pointers.  My machine seems happy with the attached patch.  Hopefully I understood enough of atomic(9).  As you suspected, it didn't work without also changing tb_ready.

Comment 31 Justin Hibbits 2024-06-04 14:42:01 UTC

Comment on attachment 251202 [details]
tbsync patch 1

All the atomic_set_*()s should really be atomic_store_*(), `atomic_set` is a bit-set (OR operation).  It's incorrect in the existing code, too.  It really doesn't make a significant difference, since the end result is the same (cpu_done should be 0 to start with, so setting nothing doesn't change that, tb_ready should be 0 to start with, so setting 1 bit does the same thing), except in the last instance to try to clear tb_ready.

In the tight loop checking if cpu_done is high enough, you don't need the acq barrier in there, because the thread fence occupying the loop does the same thing, and acq places the barrier after the operation, so in the same place anyway.

Glad to see this patch overall works for you, though!  I'm really surprised those barriers are needed, since they're not needed in the mpc85xx case, which this is derived from.

Created attachment 251226 [details]
Kernel Trap

Comment 33 Justin Hibbits 2024-06-04 20:09:37 UTC

Comment on attachment 251226 [details]
Kernel Trap

That's an illegal instruction trap, which is odd.  You'd need to 'x/i 0xc00000000a1a9c28' to see what the instruction is.

Comment on attachment 251226 [details]
Kernel Trap

Sorry, I tried to add a comment with this screenshot but it didn't quite work as expected :)

So first of all reverting c583b02587 did nothing at all as I'm observing the same symptoms as Ben.

I've managed to painfully install the lastest snapshot with a machine continually freezing (running okay for 15 secs then fans running full blast during which the computer is unresponsive for a minute of so).

Once the install completed, I did a reboot and I was welcomed by a Fatal Kernel Trap like the one I've posted, sometimes I get to see it and others the machine reboots immediately. 

I then tried to take an image from the drive with my FreeBSD PC but I can't see any partitions on it (I don't remember if the install wizard defaults to UFS or ZFS).

Please let me know if that would be of any use for troubleshooting.

I haven't yet tried Ben's patches but if it works for him there is a chance it would work for me as we're using the same HW, I'm super happy that we're making some progress !


Also there is something rather strange I've noticed with recent Images, If I boot them while holding C (cdrom) all I get is a black screen with fans roaring immediately 

but if I use the option key instead and wait for the watch cursor to turn back to a regular mouse cursor I'm able to boot just fine, I'm guessing that might be caused by a bunch of OF variables.


Thanks everyone for your efforts so far !

(In reply to Justin Hibbits from comment #33)

Yesterday I've tried Ben's patch and my machine was way happier, unfortunately the same problem still occurs, just less often.

I was able to install 14.1 and use it for a few minutes before the machine crashed while idling.

Just trying to understand why we'd want to use memory barriers with the powermac_smp_timebase_sync function.

Is it because PPC is using a weak memory model which would be problematic in an SMP context like explained below ?

https://preshing.com/20120710/memory-barriers-are-like-source-control-operations/
https://preshing.com/20120930/weak-vs-strong-memory-models/

You mentioned that the code is derived from mpc85xx, do we have such a machine in the field which is running a modern release properly ?

Also how was FreeBSD handling timebase sync before the introduction of plateform_powermac ?

I'll try your suggestions and test it again.

Comment 36 Justin Hibbits 2024-06-07 00:27:02 UTC

(In reply to Mitch from comment #35)

mpc85xx platforms I've personally tested with FreeBSD are the AmigaOne X5000 and AmigaOne A1222.  I wrote the mpc85xx timebase sync specifically for those targets, after seeing strange hanging behavior.

As for why we want to use memory barriers, we really need just execution barriers (isync), I think, because we need to make sure the timebase is correct before it's unlocked by the BSP.  This can only be ensured by using a barrier between setting the timebase and declaring done (atomic_add_int() of cpu_done).  But, yes, a weak memory model does mean we need more explicit syncs where on strong models they would be implicit (but pay the penalty on all accesses).

We really shouldn't need any syncs for tb_ready, because it can be done lazily.  The only sync we should really need is the cpu_done.

Before this timebase sync we used a synchronization mechanism at AP launch time, where we simply "hoped" that they were close enough to all get the same timebase.  This is obviously problematic.  It was changed during the 12-CURRENT time frame.  The powermac sync change was done in 2021, though.

(In reply to Justin Hibbits from comment #36)

Thanks for your detailed reply, yesterday I've installed git and tried to clone the FreeBSD source tree during which the machine crashed. 

I've archived /var/crash here, The initial crash was self induced to make sure dumping into /var/crash would work and Vmcore.1 is the Kernel crash dump which was created after my Git clone attempt. 

https://box.ochem.net/cloud/index.php/s/BEcGjstARB8bz9J