| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-15 | |||
| 17:29:50 | dansmith | oh you mean in the other repo or ours? | |
| 17:30:02 | sean-k-mooney | in the ohter repo | |
| 17:30:09 | sean-k-mooney | but if you want to test it in ours first | |
| 17:30:13 | dansmith | yeah, I don't think the owners are quite ready for that | |
| 17:30:13 | sean-k-mooney | then im ok with that | |
| 17:30:25 | dansmith | I think we should lead the way here and show that it's stable and then we can argue for it | |
| 17:30:28 | sean-k-mooney | ok just confiming | |
| 17:31:22 | sean-k-mooney | i was checkign the local.conf in tempest-integrated-compute and realised it was not set | |
| 17:31:35 | sean-k-mooney | then i realsed you only enbaled it in some of the jobs and i was trying to figure out why | |
| 17:37:28 | clarkb | dansmith: nova-next's devstack runtime was just under 11 minutes with your change. Thats pretty impressive | |
| 17:38:39 | dansmith | clarkb: yeah, I saw a super fast nova-live migration too | |
| 17:38:53 | JayF | clarkb: fyi I have https://review.opendev.org/c/openstack/ironic/+/919620 up as well, harder to tell w/ironic b/c our times vary so much per cloud | |
| 17:38:55 | sean-k-mooney | 39 mins | |
| 17:39:03 | JayF | with a ty to Dan for enabling that in Ironic | |
| 17:39:11 | sean-k-mooney | i -1 the patch because your current patch is only setting it on the conotler nodes not the compute nodes | |
| 17:39:21 | sean-k-mooney | but over all it looks good | |
| 17:40:47 | clarkb | JayF: the devstack log file logs a bunch of devstack specific runtime info at the end | |
| 17:40:57 | sean-k-mooney | well no 39 mins was the full jobs i realise now when you said 9 mints that just devstack | |
| 17:42:01 | sean-k-mooney | JayF: https://zuul.opendev.org/t/openstack/build/265ec9ab7be145ef9925f5bfef62e2f4/log/controller/logs/devstacklog.txt#21773 | |
| 17:42:04 | clarkb | JayF: for example https://zuul.opendev.org/t/openstack/build/68136706b3a446ecabfcee5e895f079c/log/controller/logs/devstacklog.txt#37755-37771 the osc line there is 2 minutes of osc but if you look at other jobs outisde of this change its like in the 5-10 minute range | |
| 17:42:52 | sean-k-mooney | clarkb: yep and the async_wait is likely reduced as well | |
| 17:43:16 | sean-k-mooney | we were likely hiding some of the latency isn async tasks | |
| 17:43:18 | clarkb | https://zuul.opendev.org/t/openstack/build/c01518a4f3a64114851dd40f7d1598e1/log/controller/logs/devstacklog.txt#31045-31061 same ironic job against a different change for comparison | |
| 17:44:29 | clarkb | 410 second difference between those two runs of the same job in the osc accounting. That is almost 7 minutes | |
| 17:44:32 | JayF | clarkb: sean-k-mooney: before: 353, after: 101 -- both on rackspace cloud | |
| 17:44:56 | sean-k-mooney | JayF: ya there is a lot of veriabity in the jobs however | |
| 17:45:05 | sean-k-mooney | but its non trivial in any case | |
| 17:45:12 | JayF | I'm sure, but 250 seconds is well beyond variance | |
| 17:45:19 | sean-k-mooney | sadly no | |
| 17:45:43 | clarkb | sean-k-mooney: in this case the low value is the indicator there is improvement | |
| 17:45:51 | clarkb | there may be that much variance but only on the high end not the low end | |
| 17:45:56 | sean-k-mooney | we see 20-30 mins of variance in some jobs depending on the provider. even within a single provier its sever mintues | |
| 17:46:11 | clarkb | particularly for osc | |
| 17:46:17 | JayF | yeah, that's what I'm saying, 100 seconds in OSC is well below the value I've seen for anything else | |
| 17:46:28 | JayF | not talking overall job run speed, just seconds in osc | |
| 17:46:33 | sean-k-mooney | clarkb: oh im not doubing this is an improvment | |
| 17:46:47 | sean-k-mooney | just that we will need more then one ci run to see how much | |
| 17:47:36 | JayF | I'm on the third example of that job and haven't found one >300 yet, in fact 353 is my lowest number yet, so I'm going to stop trying to kill my joy and say thank you for the extra 4 minutes :D | |
| 17:48:59 | sean-k-mooney | dansmith: so your change is currently only setting this on the contoler node on our multi node jobs. do we want to merge it as is and have a fix up later or fix it to apply to both. most of the destack time happens on the contoler anyway | |
| 17:49:13 | sean-k-mooney | so the speedup on the compute while non 0 will not be the main speedup in the job | |
| 17:49:18 | clarkb | I bet all of the osc stuff is on the controller too | |
| 17:49:27 | clarkb | since you interact with the apis on the controller and not compute nodes directly. | |
| 17:49:37 | sean-k-mooney | ya most if not all | |
| 17:50:00 | sean-k-mooney | actully i can check the devstack log on the compute | |
| 17:50:05 | dansmith | sean-k-mooney: I haven't looked yet, but almost if not all is on the controller, and they don't *need* to both use the server | |
| 17:51:02 | sean-k-mooney | osc 0 | |
| 17:51:06 | dansmith | yeah :) | |
| 17:51:10 | sean-k-mooney | ok so the compute does not use osc | |
| 17:51:18 | sean-k-mooney | in that case ill change my -1 to +2 | |
| 17:51:59 | dansmith | JayF: 4 minutes is about the minimum speedup I've been seeing, yeah.. up to like 6 minutes on some jobs that setup more services | |
| 17:52:33 | dansmith | ten jobs at 4 minutes each is 40 minutes of cumulative gate time which is non-trivial, IMHO | |
| 17:52:50 | JayF | 4 minutes on a job that usually takes 40-60 is amazing | |
| 17:57:35 | opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Handle MAC dash-addresses during migration https://review.opendev.org/c/openstack/nova/+/919760 | |
| 18:04:01 | dansmith | JayF: yeah, that does not suck | |
| 18:09:42 | opendevreview | ribaudr proposed openstack/nova master: Amend ShareMappingStatus due to asynchronous call https://review.opendev.org/c/openstack/nova/+/908864 | |
| 18:09:42 | opendevreview | ribaudr proposed openstack/nova master: Amend DB model add a unique constraint. https://review.opendev.org/c/openstack/nova/+/912518 | |
| 18:09:43 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | |
| 18:09:43 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | |
| 18:09:44 | opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | |
| 18:09:44 | opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | |
| 18:09:45 | opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | |
| 18:09:46 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | |
| 18:09:46 | opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | |
| 18:09:48 | opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | |
| 18:09:48 | opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | |
| 18:09:50 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | |
| 18:09:50 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part) https://review.opendev.org/c/openstack/nova/+/860287 | |
| 18:09:52 | opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860288 | |
| 18:09:52 | opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | |
| 18:09:54 | opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | |
| 18:09:54 | opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | |
| 18:09:56 | opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | |
| 18:09:56 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | |
| 18:09:58 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | |
| 18:09:58 | opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | |
| 18:10:00 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | |
| 18:10:00 | opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | |
| 18:10:02 | opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | |
| 18:10:02 | opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | |
| 18:10:04 | opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | |
| 18:16:30 | sean-k-mooney | dansmith: by the way i have like 4 other test tweek patchs that these are conficting with but ill rebase my commits after your two land | |
| 18:17:05 | dansmith | sean-k-mooney: ack, no rush on mine, but might make yours faster if you do :) | |
| 18:17:11 | dansmith | so.. | |
| 18:17:35 | dansmith | python3.12 changes the repr() format for collections.OrderedDict, which is breaking our object hash calculation (for notifications) | |
| 18:25:18 | sean-k-mooney | i added all the ci job patches im aware of to https://etherpad.opendev.org/p/nova-dalmatian-status#L17 | |
| 18:47:13 | opendevreview | Dan Smith proposed openstack/nova master: Fix notification object hashes for python 3.12 https://review.opendev.org/c/openstack/nova/+/919763 | |
| 18:47:13 | opendevreview | Dan Smith proposed openstack/nova master: Fix hacking test with syntax error https://review.opendev.org/c/openstack/nova/+/919764 | |
| 18:47:20 | dansmith | this should get the py3.12 unit job working ^ | |
| 18:51:20 | dansmith | I wonder if we could get Uggla_ to submit the manila series in smaller chunks.. I think that series is basically consuming all the nova quota in check right now | |
| 19:08:46 | opendevreview | Dan Smith proposed openstack/nova master: Make python 3.12 unit tests voting https://review.opendev.org/c/openstack/nova/+/919767 | |
| 19:10:01 | opendevreview | Dan Smith proposed openstack/nova master: Make python 3.12 unit tests voting https://review.opendev.org/c/openstack/nova/+/919767 | |
| 19:10:27 | opendevreview | Dan Smith proposed openstack/nova master: Make python 3.12 unit tests voting https://review.opendev.org/c/openstack/nova/+/919767 | |
| 19:11:28 | sean-k-mooney | dansmith: so the fix is to just bite the bullet and update the hash | |
| 19:11:57 | dansmith | sean-k-mooney: well, change the hashed things to something that is stable on both, and then yeah, update them all :( | |
| 19:11:58 | dansmith | it sucked | |
| 19:13:35 | opendevreview | Dan Smith proposed openstack/nova master: Fix notification object hashes for python 3.12 https://review.opendev.org/c/openstack/nova/+/919763 | |
| 19:13:35 | opendevreview | Dan Smith proposed openstack/nova master: Fix hacking test with syntax error https://review.opendev.org/c/openstack/nova/+/919764 | |
| 19:13:36 | opendevreview | Dan Smith proposed openstack/nova master: Make python 3.12 unit tests voting https://review.opendev.org/c/openstack/nova/+/919767 | |
| 19:14:40 | sean-k-mooney | ack so you cahnged form an ordered dict to calling sorted on the maping over the items of the schema | |
| 19:16:17 | dansmith | it was already calling sorted on the dict.items(), it was just re-dicting them with OrderedDict, which isn't necessary (dicts are ordered in python3) but even still, we don't need the extra dict layer anyway | |