| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-23 | |||
| 15:32:28 | sean-k-mooney | of swap | |
| 15:32:46 | sean-k-mooney | you could go a little lowwer but you wotn get to 2.0 with only 12G | |
| 15:33:03 | sean-k-mooney | you will hit oom issues | |
| 15:33:39 | sean-k-mooney | you have not accounted for the qemu overhead or nova/os memory usage | |
| 15:33:47 | recyclehero | good to know. I should I make changes while its a fresh deployement. | |
| 15:34:05 | recyclehero | I have counted nova/os memory in the reserverd. | |
| 15:34:24 | recyclehero | but not for the qemu overhead | |
| 15:34:51 | sean-k-mooney | the qemu over head can be small or lager depening on how my cores or devices an instance has | |
| 15:35:11 | sean-k-mooney | it can be as small as ~100MB or well over a GB | |
| 15:35:35 | sean-k-mooney | per instance | |
| 15:36:25 | sean-k-mooney | so you need to caulate what the maxium amount of overhead will be for you smallest and larges flavor by booting them then figure out how many of them will run on each host and set teh reserved value accourdignly | |
| 15:36:49 | sean-k-mooney | or allocate enough swap to copenstate | |
| 15:39:18 | recyclehero | my strategy is reserved for nova+os and swap to conpensate for overcommiting | |
| 15:40:00 | sean-k-mooney | ya but again that is ignoring qemu overhead | |
| 15:40:21 | sean-k-mooney | which is ment to be acounted for in reserved or swap | |
| 15:40:45 | sean-k-mooney | in generall i advise never overcommiting memroy or disks for what its worth | |
| 15:40:50 | sean-k-mooney | only cpus | |
| 15:41:13 | sean-k-mooney | and even then only cpus at a max of 4.0 | |
| 15:42:03 | sean-k-mooney | if you want to hevialy over commit memory you might find https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/libvirt-file-backed-memory.html intersting too | |
| 15:42:29 | sean-k-mooney | its uses files on disk that are cached in ram instead of relying on swap | |
| 15:45:31 | recyclehero | thanks sean-k-mooney for sharing ur xp | |
| 15:49:34 | sean-k-mooney | we get enough of bug reported by customer related to OOM that i always advise people against it unless the really really know what they are doing | |
| 15:50:08 | sean-k-mooney | people coming form vmware often dont know that vmware always uses file backed memroy wehn you are over commiting behind the scenes | |
| 15:58:44 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Make JSON valid https://review.opendev.org/759466 | |
| 15:58:59 | stephenfin | melwitt: lyarwood: That will fix our currently broken docs build ^ | |
| 15:59:19 | stephenfin | I don't know what changed :( Neither Sphinx nor Pygments have seen releases recently :-\ | |
| 15:59:30 | stephenfin | and that doc hasn't been changed | |
| 16:00:13 | melwitt | tbh I don't like the sound of "don't know what changed" :/ | |
| 16:01:36 | stephenfin | neither do I, but it's the best I can offer atm :) | |
| 16:01:42 | melwitt | also I don't understand how the change fixes it, what is it doing, shorten the string? | |
| 16:02:09 | stephenfin | no, what's there isn't valid JSON | |
| 16:02:25 | stephenfin | you can't escape newlines with backslash | |
| 16:02:31 | melwitt | ohh | |
| 16:02:43 | stephenfin | so the parser is rightly complaining | |
| 16:02:49 | melwitt | I see | |
| 16:02:55 | stephenfin | I just don't know why it's suddenly doing it now | |
| 16:02:58 | melwitt | so something started validating it | |
| 16:03:01 | melwitt | yeah | |
| 16:03:11 | stephenfin | but Sphinx has has this validation for years | |
| 16:03:19 | stephenfin | it's more than something has stopped ignoring that | |
| 16:03:51 | stephenfin | hmm, new release of that | |
| 16:04:58 | melwitt | yeah last time I helped with a docs job break it was bc of a new release of openstackdocstheme. so seems likely to be the cause | |
| 16:06:57 | melwitt | and I agree with you on what's likely happened, something started being strict(er) about validation | |
| 16:10:40 | stephenfin | not that, 2.2.6 hasn't made it to u-c yet | |
| 16:12:35 | stephenfin | Ahh, pygments 2.7.1 came in recently. That's our culprit (change I1bc1fe988e3cfd4824dd7b4a9327a67db46c1533) | |
| 16:14:40 | melwitt | \o/ | |
| 16:16:58 | openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Make JSON valid https://review.opendev.org/759466 | |
| 16:17:01 | stephenfin | I didn't see the change because I was searching for pygments (lower case p) :( | |
| 16:17:03 | stephenfin | vim-- | |
| 16:17:08 | stephenfin | melwitt: ^ | |
| 16:17:14 | melwitt | heh | |
| 16:17:20 | melwitt | on it | |
| 16:17:35 | stephenfin | The gate is still hopelessly broken due to this libvirt/eventlet issue but that's one thing off the list | |
| 16:20:00 | melwitt | ++ | |
| 16:20:58 | lyarwood | stephenfin: sorry wasn't watching irc upstream | |
| 16:21:05 | lyarwood | stephenfin: what's the eventlet issue? | |
| 16:21:28 | stephenfin | ||
| 16:21:28 | stephenfin | ||
| 16:21:28 | stephenfin | <stephenfin> the failing one uses eventlet===0.28.0 | |
| 16:21:28 | stephenfin | ||
| 16:21:30 | stephenfin | <stephenfin> so eventlet has borked us \o/ | |
| 16:21:55 | stephenfin | I think it's eventlet to blame, though the diff between those versions doesn't show anything particularly egregious | |
| 16:22:13 | stephenfin | mypy and greenlet were also bumped so they might be more to blame | |
| 16:33:21 | stephenfin | artom++ Good spot on https://review.opendev.org/#/c/758157/. A flaw in my testing | |
| 18:02:42 | openstackgerrit | Merged openstack/nova stable/stein: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/747611 | |
| 18:10:05 | melwitt | dansmith: if you're around, fix for the gate is at https://review.opendev.org/759466 | |
| 18:11:04 | dansmith | melwitt: ack | |
| 18:11:35 | melwitt | thanks! | |
| 18:16:15 | johnsom | Yeah, as mentioned above, nova is crashing in the Octavia gates. The traceback does point to eventlet. | |
| 18:16:16 | johnsom | https://zuul.opendev.org/t/openstack/build/2828ba97f9d54d458facd777d91dd36e/log/controller/logs/screen-n-cpu.txt#899 | |
| 18:18:56 | melwitt | urgh. so that's what stephenfin was talking about earlier. I didn't realize it was a 100% rate failure | |
| 18:20:07 | melwitt | looking into it | |
| 18:28:13 | artom | stephenfin, I do what I can:) | |
| 18:46:36 | johnsom | Here is a paste with the packages that are different between the last good run (yesterday) and the failing run today: http://paste.openstack.org/show/799337/ | |
| 18:56:23 | openstackgerrit | melanie witt proposed openstack/nova master: DNM test eventlet 0.26.1 https://review.opendev.org/759506 | |
| 18:56:37 | melwitt | ah, thank you | |
| 19:00:33 | openstackgerrit | melanie witt proposed openstack/nova master: DNM test eventlet 0.26.1 https://review.opendev.org/759506 | |
| 19:34:07 | openstackgerrit | Merged openstack/nova stable/train: compute: Validate a BDMs disk_bus when provided https://review.opendev.org/744552 | |
| 19:34:13 | openstackgerrit | Merged openstack/nova stable/ussuri: Fix invalid assert_has_calls https://review.opendev.org/751752 | |
| 19:36:53 | openstackgerrit | melanie witt proposed openstack/nova master: DNM try using autowrap_names instead of autowrap https://review.opendev.org/759513 | |
| 20:08:31 | openstackgerrit | Merged openstack/nova master: docs: Make JSON valid https://review.opendev.org/759466 | |
| 20:56:29 | openstackgerrit | Billy Olsen proposed openstack/nova stable/ussuri: add functional regression test for bug #1888395 https://review.opendev.org/759522 | |
| 20:56:30 | openstack | bug 1888395 in OpenStack Compute (nova) ussuri "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 | |
| 21:10:57 | openstackgerrit | Billy Olsen proposed openstack/nova stable/ussuri: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/759151 | |
| 23:22:04 | openstackgerrit | Billy Olsen proposed openstack/nova stable/train: add functional regression test for bug #1888395 https://review.opendev.org/759533 | |
| 23:22:05 | openstack | bug 1888395 in OpenStack Compute (nova) ussuri "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen) | |
| 23:23:24 | openstackgerrit | Billy Olsen proposed openstack/nova stable/train: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/759153 | |
| #openstack-nova - 2020-10-24 | |||
| 01:23:54 | openstackgerrit | Merged openstack/nova master: [doc]: Fix glance image_metadata link https://review.opendev.org/759100 | |
| 01:46:25 | openstackgerrit | Merged openstack/nova master: CinderFixture: Return a driver_type of `fake` within connection_info https://review.opendev.org/758238 | |
| 03:11:16 | openstackgerrit | guolei proposed openstack/nova master: Fix GPU reschedule bug https://review.opendev.org/759544 | |
| 06:12:16 | openstackgerrit | YumengBao proposed openstack/nova-specs master: SRIOV Enabled Nic Support Specification https://review.opendev.org/742785 | |
| 06:56:57 | openstackgerrit | melanie witt proposed openstack/nova master: DNM test eventlet revert from 0.28.0 to 0.26.1 https://review.opendev.org/759506 | |
| #openstack-nova - 2020-10-25 | |||
| 01:19:36 | openstackgerrit | Wenping Song proposed openstack/nova master: Remove redundant code in instance delete https://review.opendev.org/759577 | |
| 03:56:10 | openstackgerrit | Billy Olsen proposed openstack/nova stable/ussuri: add functional regression test for bug #1888395 https://review.opendev.org/759522 | |
| 03:56:11 | openstack | bug 1888395 in OpenStack Compute (nova) ussuri "shared live migration of a vm with a vif is broken in train" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen) | |
| 04:49:52 | openstackgerrit | guolei proposed openstack/nova master: Make guest's cpu pinned more balanced on each numa node https://review.opendev.org/759581 | |
| 14:46:13 | Haunted330 | This channel is closed. You are going to have to leave. | |
| #openstack-nova - 2020-10-26 | |||
| 02:18:43 | openstackgerrit | Billy Olsen proposed openstack/nova stable/ussuri: Set migrate_data.vifs only when using multiple port bindings https://review.opendev.org/759151 | |
| 08:10:29 | bauzas | good morning Nova | |
| 08:57:08 | gibi | bauzas: good mornin | |