Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x86{,_64} build hangs after next-20181008 #213

Closed
nathanchance opened this issue Oct 11, 2018 · 16 comments
Closed

x86{,_64} build hangs after next-20181008 #213

nathanchance opened this issue Oct 11, 2018 · 16 comments
Assignees
Labels
[ARCH] x86 This bug impacts ARCH=i386 [ARCH] x86_64 This bug impacts ARCH=x86_64 [BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 4.20 This bug was fixed in Linux 4.20

Comments

@nathanchance
Copy link
Member

Can be reproduced with defconfig with or without CONFIG_64BIT set:

  CC      drivers/tty/tty_audit.o
  AR      drivers/video/fbdev/core/built-in.a
  CC      drivers/gpu/drm/drm_debugfs_crc.o
  CC      drivers/gpu/drm/drm_debugfs.o
  CC      drivers/tty/sysrq.o
  AR      drivers/video/fbdev/built-in.a
  CC      drivers/gpu/drm/drm_mipi_dsi.o
  CC      drivers/gpu/drm/drm_panel_orientation_quirks.o
  AR      drivers/video/built-in.a
  AR      drivers/tty/built-in.a
  AR      drivers/gpu/drm/built-in.a
  AR      drivers/gpu/built-in.a
  AR      drivers/built-in.a

It seems to be stuck at init/version.o, as I see this when I interrupt the build:

^Cmake[1]: *** Deleting file 'init/version.o'
make[1]: *** [scripts/Makefile.build:305: init/version.o] Interrupt
make: *** [Makefile:1050: init] Interrupt

Bisect log:

git bisect start
# good: [64c5e530ac2cab7e710c3bd08ef8594a016e4f9a] Merge tag 'arc-4.19-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/vgupta/arc
git bisect good 64c5e530ac2cab7e710c3bd08ef8594a016e4f9a
# bad: [7f3049305d22fc543f836d00aeea41ecef911199] Add linux-next specific files for 20181010
git bisect bad 7f3049305d22fc543f836d00aeea41ecef911199
# good: [6d67f8d0eec9b0b3234e77ece3465361c19f7d3c] Merge remote-tracking branch 'spi-nor/spi-nor/next'
git bisect good 6d67f8d0eec9b0b3234e77ece3465361c19f7d3c
# bad: [6a2e03f7dd9fec432789e77c34743c8773155461] Merge remote-tracking branch 'tip/auto-latest'
git bisect bad 6a2e03f7dd9fec432789e77c34743c8773155461
# good: [f0502f1777c3a11a9b7bbf93cbde467051447830] Merge remote-tracking branch 'sound/for-next'
git bisect good f0502f1777c3a11a9b7bbf93cbde467051447830
# good: [0f234d9a5ab00374969b5e271be40b429cb1d9d8] Merge remote-tracking branch 'iommu/next'
git bisect good 0f234d9a5ab00374969b5e271be40b429cb1d9d8
# good: [fac07d2ba7b2764e3002ff9bc7861742a84a2ef6] Merge branch 'perf/core'
git bisect good fac07d2ba7b2764e3002ff9bc7861742a84a2ef6
# bad: [d74865bd3996c7a6f3e8ce6e626c1fe474e39494] Merge branch 'x86/mm'
git bisect bad d74865bd3996c7a6f3e8ce6e626c1fe474e39494
# good: [dcd2d0cece1608b2be9184786c900807ec947076] Merge branch 'x86/asm'
git bisect good dcd2d0cece1608b2be9184786c900807ec947076
# bad: [ae9260d80e517c8702b91b8e00d117e1e2834c33] Merge branch 'x86/cache'
git bisect bad ae9260d80e517c8702b91b8e00d117e1e2834c33
# bad: [d5a581d84ae6b8a4a740464b80d8d9cf1e7947b2] x86/cpufeature: Macrofy inline assembly code to work around GCC inlining bugs
git bisect bad d5a581d84ae6b8a4a740464b80d8d9cf1e7947b2
# good: [c0554d2d3db438623b4f2f9abc3d766b2b15d2fb] Merge branch 'linus' into x86/core, to pick up fixes
git bisect good c0554d2d3db438623b4f2f9abc3d766b2b15d2fb
# bad: [77f48ec28e4ccff94d2e5f4260a83ac27a7f3099] x86/alternatives: Macrofy lock prefixes to work around GCC inlining bugs
git bisect bad 77f48ec28e4ccff94d2e5f4260a83ac27a7f3099
# bad: [77b0bf55bc675233d22cd5df97605d516d64525e] kbuild/Makefile: Prepare for using macros in inline assembly code to work around asm() related GCC inlining bugs
git bisect bad 77b0bf55bc675233d22cd5df97605d516d64525e
# good: [35e76b99ddf20405a6196bb7c9eb152675c93106] kbuild/arch/xtensa: Define LINKER_SCRIPT for the linker script
git bisect good 35e76b99ddf20405a6196bb7c9eb152675c93106
# first bad commit: [77b0bf55bc675233d22cd5df97605d516d64525e] kbuild/Makefile: Prepare for using macros in inline assembly code to work around asm() related GCC inlining bugs

Bad commit: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=77b0bf55bc675233d22cd5df97605d516d64525e

I applied @bulwahn's three patches in #186 to make sure none of my own patches were causing any issues.

@nathanchance nathanchance added [BUG] Untriaged Something isn't working [ARCH] x86_64 This bug impacts ARCH=x86_64 [ARCH] x86 This bug impacts ARCH=i386 labels Oct 11, 2018
@nickdesaulniers
Copy link
Member

Does make V=1 provide more info?

@nathanchance
Copy link
Member Author

I don't think so, here's the last bit I see

   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/tileblit.o";
   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/cfbcopyarea.o";   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/cfbimgblt.o";
   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/sysfillrect.o";   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/sysimgblt.o";
   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/fbcon.o";
   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/fb_sys_fops.o";   ./tools/objtool/objtool orc generate  --no-fp --retpoline "drivers/video/fbdev/core/syscopyarea.o";  rm -f drivers/video/fbdev/core/built-in.a; ar rcSTPD drivers/video/fbdev/core/built-in.a drivers/video/fbdev/core/fb_cmdline.o drivers/video/fbdev/core/fb_notify.o drivers/video/fbdev/core/fbmem.o drivers/video/fbdev/core/fbmon.o drivers/video/fbdev/core/fbcmap.o drivers/video/fbdev/core/fbsysfs.o drivers/video/fbdev/core/modedb.o drivers/video/fbdev/core/fbcvt.o drivers/video/fbdev/core/fb_defio.o drivers/video/fbdev/core/fbcon.o drivers/video/fbdev/core/bitblit.o drivers/video/fbdev/core/softcursor.o
drivers/video/fbdev/core/tileblit.o drivers/video/fbdev/core/cfbfillrect.o drivers/video/fbdev/core/cfbcopyarea.o drivers/video/fbdev/core/cfbimgblt.o drivers/video/fbdev/core/sysfillrect.o drivers/video/fbdev/core/syscopyarea.o drivers/video/fbdev/core/sysimgblt.o drivers/video/fbdev/core/fb_sys_fops.o  (cat /dev/null;   cat drivers/video/fbdev/core/modules.order;   cat drivers/video/fbdev/omap2/modules.order;) > drivers/video/fbdev/modules.order                                                             rm -f drivers/video/fbdev/built-in.a; ar rcSTPD drivers/video/fbdev/built-in.a drivers/video/fbdev/core/built-in.a drivers/video/fbdev/omap2/built-in.a drivers/video/fbdev/efifb.o                       (cat /dev/null;   cat drivers/video/console/modules.order;   cat drivers/video/logo/modules.order;   cat drivers/video/backlight/modules.order;   cat drivers/video/fbdev/modules.order;) > drivers/video/modules.order                                                                                             rm -f drivers/video/built-in.a; ar rcSTPD drivers/video/built-in.a drivers/video/hdmi.o drivers/video/console/built-in.a drivers/video/logo/built-in.a drivers/video/backlight/built-in.a drivers/video/fbdev/built-in.a                                                                                        (cat /dev/null;   cat drivers/irqchip/modules.order;   cat drivers/bus/modules.order;   cat drivers/pwm/modules.order;   cat drivers/pci/modules.order;   cat drivers/video/modules.order;   cat drivers/idle/modules.order;   cat drivers/char/ipmi/modules.order;   cat drivers/acpi/modules.order;   cat drivers/pnp/modules.order;   cat drivers/amba/modules.order;   cat drivers/clk/modules.order;   cat drivers/dma/modules.order;   cat drivers/soc/modules.order;   cat drivers/tty/modules.order;   cat drivers/char/modules.order;   cat drivers/iommu/modules.order;   cat drivers/gpu/modules.order;   cat drivers/connector/modules.order;   cat drivers/base/modules.order;   cat drivers/block/modules.order;   cat drivers/misc/modules.order;   cat drivers/mfd/modules.order;   cat drivers/nfc/modules.order;   cat drivers/dma-buf/modules.order;   cat drivers/macintosh/modules.order;   cat drivers/scsi/modules.order;   cat drivers/nvme/modules.order;   cat drivers/ata/modules.order;   cat drivers/net/modules.order;   cat drivers/firewire/modules.order;   cat drivers/cdrom/modules.order;   cat drivers/auxdisplay/modules.order;   cat drivers/pcmcia/modules.order;   cat drivers/usb/modules.order;   cat drivers/usb/modules.order;   cat drivers/usb/modules.order;   cat drivers/input/serio/modules.order;   cat drivers/input/modules.order;   cat drivers/rtc/modules.order;   cat drivers/i2c/modules.order;   cat drivers/media/modules.order;   cat drivers/pps/modules.order;   cat drivers/ptp/modules.order;   cat drivers/power/modules.order;   cat drivers/hwmon/modules.order;   cat drivers/thermal/modules.order;   cat drivers/watchdog/modules.order;   cat drivers/md/modules.order;   cat drivers/edac/modules.order;   cat drivers/cpufreq/modules.order;   cat drivers/cpuidle/modules.order;   cat drivers/mmc/modules.order;   cat drivers/leds/modules.order;   cat drivers/firmware/modules.order;   cat drivers/crypto/modules.order;   cat drivers/clocksource/modules.order;   cat drivers/hid/modules.order;   cat drivers/platform/modules.order;   cat drivers/mailbox/modules.order;   cat drivers/perf/modules.order;   cat drivers/ras/modules.order;   cat drivers/hwtracing/intel_th/modules.order;   cat drivers/nvmem/modules.order;) > drivers/modules.order
  rm -f drivers/built-in.a; ar rcSTPD drivers/built-in.a drivers/irqchip/built-in.a drivers/bus/built-in.a drivers/pwm/built-in.a drivers/pci/built-in.a drivers/video/built-in.a drivers/idle/built-in.a drivers/char/ipmi/built-in.a drivers/acpi/built-in.a drivers/pnp/built-in.a drivers/amba/built-in.a drivers/clk/built-in.a drivers/dma/built-in.a drivers/soc/built-in.a drivers/tty/built-in.a drivers/char/built-in.a drivers/iommu/built-in.a drivers/gpu/built-in.a drivers/connector/built-in.a drivers/base/built-in.a drivers/block/built-in.a drivers/misc/built-in.a drivers/mfd/built-in.a drivers/nfc/built-in.a drivers/dma-buf/built-in.a drivers/macintosh/built-in.a drivers/scsi/built-in.a drivers/nvme/built-in.a drivers/ata/built-in.a drivers/net/built-in.a drivers/firewire/built-in.a drivers/cdrom/built-in.a drivers/auxdisplay/built-in.a drivers/pcmcia/built-in.a drivers/usb/built-in.a drivers/input/serio/built-in.a drivers/input/built-in.a drivers/rtc/built-in.a drivers/i2c/built-in.a drivers/media/built-in.a drivers/pps/built-in.a drivers/ptp/built-in.a drivers/power/built-in.a drivers/hwmon/built-in.a drivers/thermal/built-in.a drivers/watchdog/built-in.a drivers/md/built-in.a drivers/edac/built-in.a drivers/cpufreq/built-in.a drivers/cpuidle/built-in.a drivers/mmc/built-in.a drivers/leds/built-in.a drivers/firmware/built-in.a drivers/crypto/built-in.a drivers/clocksource/built-in.a drivers/hid/built-in.a drivers/platform/built-in.a drivers/mailbox/built-in.a drivers/perf/built-in.a drivers/ras/built-in.a drivers/hwtracing/intel_th/built-in.a drivers/nvmem/built-in.a

@nickdesaulniers
Copy link
Member

so ar is the last thing being run? Can you make sure you're not building with -j<#> to be sure ar is the final command? Does top show some build utility or clang pegging a CPU at full throttle?

@nathanchance
Copy link
Member Author

I've gone ahead and compiled with make -j1 to make sure it was only executing one command at a time and it looks like it's hung on init/version.o: https://gist.github.com/nathanchance/c52b8235b8fe4c1b2568dad82b19c6e8

Here is a screenshot of htop:

screenshot 2018-10-11 at 11 38 46

@nathanchance
Copy link
Member Author

nathanchance commented Oct 11, 2018

Looks like removing -Wa,- from arch/x86/Makefile fixes it. Wonder what that flag even means or if it is left over from an earlier patchset.

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 73f4831283ac..4d3612ad0f77 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -239,7 +239,7 @@ archheaders:
 archmacros:
        $(Q)$(MAKE) $(build)=arch/x86/kernel arch/x86/kernel/macros.s
 
-ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
+ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s
 export ASM_MACRO_FLAGS
 KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)
 

@nickdesaulniers
Copy link
Member

From your screenshot, you can see that make has forked a child make which has forked a ccache which has forked a clang which has forked an as for producing -o init/main.o. The rest is cutoff, but I assume - is being passed to GAS?

I have no idea what -Wa,- means to the assembler. Maybe read from stdin? Does your patch break the gcc build? If it does not, then it may be safe to remove. (Nice find, btw).

@nathanchance
Copy link
Member Author

Yeah I think you are right, I'll be able to confirm later.

Unfortunately, GCC completely blows up with that diff but it's entirely possible that flag just needs to be added if Clang isn't being used. I'll see if I can get qemu set up to boot a kernel compiled with Clang + that diff and see if anything explodes.

@nathanchance
Copy link
Member Author

diff --git a/arch/x86/Makefile b/arch/x86/Makefile
index 73f4831283ac..be8be3dd38b1 100644
--- a/arch/x86/Makefile
+++ b/arch/x86/Makefile
@@ -239,7 +239,10 @@ archheaders:
 archmacros:
        $(Q)$(MAKE) $(build)=arch/x86/kernel arch/x86/kernel/macros.s

-ASM_MACRO_FLAGS = -Wa,arch/x86/kernel/macros.s -Wa,-
+ASM_MACRO_FLAGS := -Wa,arch/x86/kernel/macros.s
+ifneq ($(cc-name),clang)
+ASM_MACRO_FLAGS += -Wa,-
+endif
 export ASM_MACRO_FLAGS
 KBUILD_CFLAGS += $(ASM_MACRO_FLAGS)

I can build and boot this diff into a Debian shell in QEMU with both GCC 8.2.1 and Clang 8 (defconfig + kvmconfig). No idea if that is a proper fix or not but it's something I can roll with for further testing.

@nickdesaulniers
Copy link
Member

I too could repro locally. That change looks good, but I kind of want to ask the author what that's doing, since I don't understand it.

@nathanchance
Copy link
Member Author

nathanchance commented Oct 22, 2018

Yeah I think I'll send it as an RFC now just to see what they say. My guess is it has something to do with how Clang and GCC fork off gas.

@nathanchance
Copy link
Member Author

@nickdesaulniers
Copy link
Member

nickdesaulniers commented Oct 23, 2018

Filed a bug to investigate this on the Clang side: https://llvm.org/pr39410

@nathanchance
Copy link
Member Author

Isn't it expected that passing -Wa,- without -pipe would cause the build to hang (because Clang is generating temporary files but gas is expecting a pipe)? On the other hand, if the build hangs with -pipe and -Wa,-, it seems to me that could be a bug.

@nathanchance nathanchance added [BUG] linux A bug that should be fixed in the mainline kernel. [BUG] llvm A bug that should be fixed in upstream LLVM [PATCH] Submitted A patch has been submitted for review and removed [BUG] Untriaged Something isn't working [BUG] llvm A bug that should be fixed in upstream LLVM labels Oct 23, 2018
@nathanchance
Copy link
Member Author

@nathanchance nathanchance self-assigned this Oct 24, 2018
fengguang pushed a commit to 0day-ci/linux that referenced this issue Oct 24, 2018
Commit 77b0bf5 ("kbuild/Makefile: Prepare for using macros in
inline assembly code to work around asm() related GCC inlining bugs")
added -Wa,- to KBUILD_CFLAGS, which breaks compiling with Clang (hangs
indefinitely at compiling init/main.o). This happens because while Clang
accepts -pipe (and has it documented in its list of supported flags), it
silently ignores it after this 2010 commit (thanks to Nick Desaulniers
for tracking this down), meaning that gas just infinitely waits for
stdin and never receives it.

llvm-mirror/clang@c19a12d

Initially, I had suggested just add -Wa,- to KBUILD_CFLAGS when GCC was
being used but that was before realizing it is because Clang doesn't do
anything with -pipe. H. Peter Anvin suggested checking to see if -pipe
gives us any gains out of GCC. Turns out it might actually be hurting:

With -pipe:

real    3m40.813s
real    3m44.449s
real    3m39.648s

Without -pipe:

real    3m38.492s
real    3m38.335s
real    3m38.975s

The issue of -Wa,- being passed along to gas without -pipe being
supported should still probably be fixed on the LLVM side (open issue:
https://bugs.llvm.org/show_bug.cgi?id=39410) but this is not as much of
a workaround anymore since it helps both GCC and Clang.

Link: ClangBuiltLinux#213
Suggested-by: H. Peter Anvin <hpa@zytor.com>
Signed-off-by: Nathan Chancellor <natechancellor@gmail.com>
@nathanchance
Copy link
Member Author

@nathanchance nathanchance added [PATCH] Accepted A submitted patch has been accepted upstream and removed [PATCH] Submitted A patch has been submitted for review labels Nov 5, 2018
@nathanchance
Copy link
Member Author

Merged and released in 4.20-rc2: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=437e88ab8f9e2ad90576ab74c4cf8f527bbf51cd

@nathanchance nathanchance added the [FIXED][LINUX] 4.20 This bug was fixed in Linux 4.20 label Nov 12, 2018
@nathanchance nathanchance removed the [PATCH] Accepted A submitted patch has been accepted upstream label Nov 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[ARCH] x86 This bug impacts ARCH=i386 [ARCH] x86_64 This bug impacts ARCH=x86_64 [BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 4.20 This bug was fixed in Linux 4.20
Projects
None yet
Development

No branches or pull requests

2 participants