Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix linux ci #1

Closed
wants to merge 1 commit into from
Closed

Fix linux ci #1

wants to merge 1 commit into from

Conversation

ahornby
Copy link
Owner

@ahornby ahornby commented Sep 19, 2023

No description provided.

@ahornby ahornby force-pushed the fix_linux_ci branch 5 times, most recently from 5f81e15 to 4fad64e Compare September 23, 2023 11:34
@ahornby ahornby force-pushed the fix_linux_ci branch 4 times, most recently from d874331 to 45a196c Compare October 1, 2023 17:19
@ahornby ahornby force-pushed the fix_linux_ci branch 2 times, most recently from 239d51c to 2d5a0e9 Compare October 13, 2023 16:05
@ahornby ahornby force-pushed the fix_linux_ci branch 3 times, most recently from 510c685 to efafb28 Compare October 20, 2023 17:11
@ahornby ahornby force-pushed the fix_linux_ci branch 4 times, most recently from eb2627d to fc93d98 Compare November 3, 2023 11:59
@ahornby ahornby force-pushed the fix_linux_ci branch 4 times, most recently from 4eaa0d0 to fe86669 Compare December 3, 2023 21:09
@ahornby ahornby force-pushed the fix_linux_ci branch 4 times, most recently from 6688d48 to 90bcf75 Compare December 16, 2023 16:35
@ahornby ahornby force-pushed the fix_linux_ci branch 2 times, most recently from 040d248 to dc1930e Compare February 15, 2024 20:13
ahornby pushed a commit that referenced this pull request May 11, 2024
@skrueger @facebook-github-bot
Fix TSAN data race in ThreadLocalDetailTest.MultiThreadedTest
58e8d1a 
Summary:
ThreadLocalDetailTest.MultiThreadedTest had a data race on std::vector because multiple threads were accessing and modifying the std::vector without a mutex. I update the code to use indexes such that the vector is not updated concurrently.

Previous run failure that this fixes:

```
stderr:
==================
WARNING: ThreadSanitizer: data race (pid=1249005)
  Write of size 8 at 0x7fff195936d0 by main thread:
    #0 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::pop_back() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1228 (thread_local_detail_test+0x116c28)
    #1 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody() fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:121 (thread_local_detail_test+0x101aba)
    facebook#2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xacf3c)
    facebook#3 testing::Test::Run() fbsource/src/gtest.cc:2692 (libthird-party_googletest_1.14.0_gtest.so+0xacb9c)
    facebook#4 testing::TestInfo::Run() fbsource/src/gtest.cc:2841 (libthird-party_googletest_1.14.0_gtest.so+0xafa4a)
    facebook#5 testing::TestSuite::Run() fbsource/src/gtest.cc:3020 (libthird-party_googletest_1.14.0_gtest.so+0xb4303)
    facebook#6 testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925 (libthird-party_googletest_1.14.0_gtest.so+0xd105e)
    facebook#7 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xd08f1)
    facebook#8 testing::UnitTest::Run() fbsource/src/gtest.cc:5489 (libthird-party_googletest_1.14.0_gtest.so+0xd037e)
    facebook#9 RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317 (libcommon_gtest_light_main.so+0x22a7)
    facebook#10 main fbcode/common/gtest/LightMain.cpp:20 (libcommon_gtest_light_main.so+0x2131)

  Previous read of size 8 at 0x7fff195936d0 by thread T123:
    #0 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::size() const fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:919 (thread_local_detail_test+0x11b0bd)
    #1 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::operator[](unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1045 (thread_local_detail_test+0x11d101)
    facebook#2 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0::operator()() const fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:97 (thread_local_detail_test+0x11d08c)
    facebook#3 void std::__invoke_impl<void, folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>(std::__invoke_other, folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:61 (thread_local_detail_test+0x11cf95)
    facebook#4 std::__invoke_result<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>::type std::__invoke<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>(folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:96 (thread_local_detail_test+0x11cf05)
    facebook#5 void std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:253 (thread_local_detail_test+0x11cebd)
    facebook#6 std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>::operator()() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:260 (thread_local_detail_test+0x11ce65)
    facebook#7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>>::_M_run() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:211 (thread_local_detail_test+0x11cd79)
    facebook#8 execute_native_thread_routine /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82:18 (libstdc++.so.6+0xdf4e4) (BuildId: 452d1cdae868baeeb2fdf1ab140f1c219bf50c6e)

  Location is stack of main thread.

  Location is global '??' at 0x7fff19575000 ([stack]+0x1e6d0)

  Thread T123 (tid=1249233, running) created by main thread at:
    #0 pthread_create <null> (thread_local_detail_test+0x156bef)
    #1 __gthread_create /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/include/x86_64-facebook-linux/bits/gthr-default.h:663:35 (libstdc++.so.6+0xdf80e) (BuildId: 452d1cdae868baeeb2fdf1ab140f1c219bf50c6e)
    facebook#2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:147:37 (libstdc++.so.6+0xdf80e)
    facebook#3 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody() fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:93 (thread_local_detail_test+0x1015cd)
    facebook#4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xacf3c)
    facebook#5 testing::Test::Run() fbsource/src/gtest.cc:2692 (libthird-party_googletest_1.14.0_gtest.so+0xacb9c)
    facebook#6 testing::TestInfo::Run() fbsource/src/gtest.cc:2841 (libthird-party_googletest_1.14.0_gtest.so+0xafa4a)
    facebook#7 testing::TestSuite::Run() fbsource/src/gtest.cc:3020 (libthird-party_googletest_1.14.0_gtest.so+0xb4303)
    facebook#8 testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925 (libthird-party_googletest_1.14.0_gtest.so+0xd105e)
    facebook#9 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xd08f1)
    facebook#10 testing::UnitTest::Run() fbsource/src/gtest.cc:5489 (libthird-party_googletest_1.14.0_gtest.so+0xd037e)
    facebook#11 RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317 (libcommon_gtest_light_main.so+0x22a7)
    facebook#12 main fbcode/common/gtest/LightMain.cpp:20 (libcommon_gtest_light_main.so+0x2131)

ThreadSanitizer: data race fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1228 in std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::pop_back()
==================
ThreadSanitizer: reported 1 warnings
```

Reviewed By: yfeldblum

Differential Revision: D56643303

fbshipit-source-id: cc364817ae79bc6418cc07faa35e1bcd21830c66
@ahornby
[getdeps] restore rpm system deps for fedora
b13758b 
Summary:

centOS (even stream) tends to be quite old and thus ends up having its rpm system deps turned off, e.g. boost in D55758008 and glog and gtest in D51813855.  Unfortunate the rpm deps were removed entirely rather than limited by distro or distro_vers.

Lets renable rpm deps but specify distro=fedora to reduce the risk of people removing them entirely.

Also:
   * fedora has up to date gtest & gmock, so use them
   * zlib-ng-compat added from fedora 38 with zlib removed from F40, so update zlib rpm names.  F37 is EOL

Test Plan:

Build locally on fedora 40:
```
./build/fbcode_builder/getdeps.py install-system-deps --recursive folly
./build/fbcode_builder/getdeps.py --allow-system-packages build --src-dir=. folly
```
@ahornby ahornby closed this Sep 19, 2024
ahornby pushed a commit that referenced this pull request Oct 18, 2024
@DenisYaroshevskiy @facebook-github-bot
migrate simdForEach #1
04dc57e 
Summary:
Moving simdForEachAligning into folly/algorithm/simd.

Step 1. Just move the files.

Reviewed By: dmm-fb

Differential Revision: D59680837

fbshipit-source-id: ad04d5a881bf8dbda186d1fd5cb943dae011e7ac
ahornby pushed a commit that referenced this pull request Oct 18, 2024
@DenisYaroshevskiy @facebook-github-bot
folly concepts #1 (facebook#2249)
dc875e5 
Summary:
Pull Request resolved: facebook#2249

We have a need for some common concepts that are used in many places.

This is a proposal, please let me know what you think.

Usages example:

`uncvref_same_as`:

https://www.internalfb.com/code/fbsource/[a71ce59323a15fd7559d6132c1f8e2d0256ca7cb]/fbcode/multifeed/entrepot/lib/zdb_manager/ZdbUtil.h?lines=237

`uncvref_instantiated_from`:

https://www.internalfb.com/code/fbsource/[b366209d2c108133a0545edc7943b9461323623b]/fbcode/multifeed/mfaction/MFHashSerializer.h?lines=38

Reviewed By: yfeldblum

Differential Revision: D59396081

fbshipit-source-id: 4912c2f9fe683f4d3b37fce43466c9281d03ce13
ahornby pushed a commit that referenced this pull request Oct 18, 2024
@spikeh @facebook-github-bot
folly: io_uring: only call readCallback if it did not change
7ac9203 
Summary:
Ucache with io_uring is crashing with this stack trace:

```
(lldb) bt
* thread #1, name = 'ucache', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x0000000000000000
    frame #1: 0x00007f517ba44560 libc.so.6`__restore_rt at libc_sigaction.c:13
    frame facebook#2: 0x000000000a16e249 ucache`folly::AsyncIoUringSocket::ReadSqe::callback(this=0x00007f498c5955e0, cqe=<unavailable>) at AsyncIoUringSocket.cpp:639
    frame facebook#3: 0x000000000964196a ucache`folly::IoUringBackend::getActiveEvents(folly::IoUringBackend::WaitForEventsMode) [inlined] folly::IoSqeBase::internalCallback(this=<unavailable>, cqe=<unavailable>) at IoUringBackend.cpp:0
    frame facebook#4: 0x000000000964195c ucache`folly::IoUringBackend::getActiveEvents(folly::IoUringBackend::WaitForEventsMode) [inlined] folly::IoUringBackend::internalProcessCqe(this=0x00007f4a8f989c00, maxGet=4294967295, mode=NORMAL) at IoUringBackend.cpp:1556
    frame facebook#5: 0x000000000964190f ucache`folly::IoUringBackend::getActiveEvents(this=0x00007f4a8f989c00, waitForEvents=<unavailable>) at IoUringBackend.cpp:1527
    frame facebook#6: 0x0000000009640ac9 ucache`folly::IoUringBackend::eb_event_base_loop(this=0x00007f4a8f989c00, flags=1) at IoUringBackend.cpp:1178
    frame facebook#7: 0x000000001b61547c ucache`folly::EventBase::loopMain(this=0x00007f4a8f97af00, flags=0, options=<unavailable>) at EventBase.cpp:0
    frame facebook#8: 0x000000001bab9a2d ucache`folly::EventBase::loopBody(this=0x00007f4a8f97af00, flags=0, options=(ignoreKeepAlive = false, allowSuspension = false)) at EventBase.cpp:513
    frame facebook#9: 0x000000000a3a6c90 ucache`folly::EventBase::loop(this=<unavailable>) at EventBase.cpp:474
    frame facebook#10: 0x000000000a3a75f7 ucache`folly::EventBase::loopForever(this=<unavailable>) at EventBase.cpp:781
    frame facebook#11: 0x000000000a436907 ucache`folly::IOThreadPoolExecutor::threadRun(this=0x00007f4da3fce1c0, thread=<unavailable>) at IOThreadPoolExecutor.cpp:240
    frame facebook#12: 0x0000000009ed68b8 ucache`void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>((null)=<unavailable>, __f=<unavailable>, __t=<unavailable>, __args=<unavailable>) at invoke.h:74
    frame facebook#13: 0x00007f517b6df4e5 libstdc++.so.6`std::execute_native_thread_routine(__p=0x00007f4da01bdbe0) at thread.cc:82:18
    frame facebook#14: 0x00007f517ba9abc9 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:434:8
    frame facebook#15: 0x00007f517bb2cf5c libc.so.6`__clone3 at clone3.S:81
```

In frame 2 the offending function is `ReadSqe::callback()`: https://fburl.com/code/4fb46hdq

The problem is that `readCallback_` is null:

```
(lldb) fr v readCallback_
(folly::AsyncReader::ReadCallback *) readCallback_ = nullptr
```

However, `readCallback_` is checked earlier in the the function `ReadSqe::callback()`: https://fburl.com/code/dhnndk29

`AsyncIoUringSocket::readError()` now fails all pending writes and calls `writeErr()`, whose implementation can then close out a socket and call `AsyncIoUringSocket::setReadCB(nullptr)` to change the callback.

Move the call to `AsyncIoUringSocket::readError()` after the call to `readCallback_`.

Reviewed By: dmm-fb

Differential Revision: D60880377

fbshipit-source-id: 42d2f430cd3dc3f4787d0e4c29da7c8ef31ba9a8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ahornby