| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-25 | |||
| 16:09:38 | efried | It would be, like, two of you using it? | |
| 16:11:09 | stephenfin | Perhaps, but it just rubs me the wrong way if we're being honest | |
| 16:11:30 | stephenfin | I'm not sure why we should be so averse to trying new things, especially when those things are opt-in and don't affect the end product in any way | |
| 16:11:35 | Nick_A | what is the correct way to enable maintenance mode on a hypervisor to prevent new instances from spawning on it? https://docs.openstack.org/python-openstackclient/rocky/cli/command-objects/host.html doesn't seem to work - "Not Implemented" error | |
| 16:12:31 | Nick_A | Never mind - we found it | |
| 16:12:34 | artom | cdent, https://review.opendev.org/#/c/644793/12/nova/tests/functional/libvirt/test_numa_servers.py culprit found | |
| 16:12:52 | cdent | woot | |
| 16:12:53 | artom | cdent, thanks again for your prompt help! | |
| 16:13:19 | cdent | you're welcome. was there a line number associated with that link? | |
| 16:13:43 | cdent | or just the general concept | |
| 16:14:12 | artom | cdent, in general - more specifically, it removed a monkey patch in setup: https://review.opendev.org/#/c/644793/12/nova/tests/functional/libvirt/base.py | |
| 16:14:23 | artom | So then had to mock a bunch of stuff for each test | |
| 16:14:30 | cdent | ah ha | |
| 16:14:50 | artom | Which I wasn't mocking, as I wrote the test before that landed | |
| 16:15:08 | artom | And thus was relying on the setUp monkeypatch, which got pulled from under me | |
| 16:15:19 | cdent | yowsa | |
| 16:15:51 | artom | TBH, the commit message doensn't do a great job of explaining *why* it was necessary to remove that monkeypatch | |
| 16:15:57 | stephenfin | idk. I do a lot of reviews. I write some code. I'm a decent community member, in general. Why do I have to pull teeth to get something in that I'm saying helps my productivity and doesn't hamper anyone else. It's frustrating. | |
| 16:16:04 | artom | But I assume there's a larger context that I'm not aware of | |
| 16:16:36 | efried | stephenfin: +2 on the basis of you really wanting it. | |
| 16:16:59 | stephenfin | Cheers | |
| 16:17:08 | efried | stephenfin: but to answer your question, because it's trivial for you to do it locally without "polluting" the nova codebase with something that's irrelevant to nova. | |
| 16:17:16 | cdent | stephenfin++ | |
| 16:17:49 | stephenfin | My counterpoint to that is that I've done, and continue to do, a lot of unpolluting | |
| 16:17:51 | efried | be like me putting a pycharm config | |
| 16:18:25 | artom | Sounds like we need the equivalent of a carbon tax | |
| 16:18:28 | efried | big difference between obsolete nova code and something that was never relevant to nova. | |
| 16:19:46 | stephenfin | relevant to nova developers though | |
| 16:20:03 | stephenfin | who are as important, if not more important, than the code | |
| 16:20:12 | efried | as relevant to any project's developers, nah? | |
| 16:20:43 | efried | are you going on a crusade to propose this same thing to all projects you work on? | |
| 16:21:08 | artom | cdent, woot, I'm back to last night's failure | |
| 16:21:19 | cdent | something reasonable now | |
| 16:21:20 | artom | *tackling | |
| 16:21:35 | stephenfin | I'll probably add it to one or two of my personal projects, maybe Sphinx too, but I wouldn't be touching oslo and the likes, no | |
| 16:21:37 | kashyap | artom: Hehe, one letter changes the meaning, doesn't it :D | |
| 16:21:59 | stephenfin | Basically anywhere where I'm likely to be undertaking large feature work consisting of many patches | |
| 16:22:23 | artom | kashyap, at least I wasn't tickling it | |
| 16:23:15 | kashyap | mriedem: Try this, not sure if that's your glass of (root?) beer -- https://www.youtube.com/watch?v=jBo870lVUyc | |
| 16:23:57 | kashyap | [Preferably with a good quality headset / speaker] | |
| 16:27:26 | cdent | oh. that's nice. | |
| 16:29:15 | kashyap | Jimmy Smith++ | |
| 17:02:46 | mriedem | what in tarnations, created devstack from master today, create a server, n-cpu logs say the guest was created in the hypervisor, and then things just hang - and virsh list doesn't show anything | |
| 17:02:47 | mriedem | wtf | |
| 17:05:36 | sean-k-mooney | mriedem: im guessing libvirt crahsed | |
| 17:05:57 | sean-k-mooney | either that or you need to run virsh listh with either sudo or --all | |
| 17:06:19 | sean-k-mooney | actully if it hung then ingore the last bit | |
| 17:06:26 | mriedem | oh right sudo virsh list | |
| 17:06:31 | mriedem | libvirtd is green | |
| 17:06:46 | mriedem | the domain is just hung in paused state | |
| 17:07:27 | sean-k-mooney | the domain or the nova compute agent | |
| 17:07:38 | mriedem | the domain | |
| 17:07:47 | mriedem | $ sudo virsh list --all | |
| 17:07:47 | mriedem | Id Name State | |
| 17:07:47 | mriedem | ---------------------------------------------------- | |
| 17:07:47 | mriedem | 3 instance-00000003 paused | |
| 17:08:03 | sean-k-mooney | and in nova its active | |
| 17:08:09 | mriedem | no | |
| 17:08:20 | mriedem | it's building b/c the libvirt driver is waiting for the power state to change from paused to running | |
| 17:09:30 | sean-k-mooney | oh and you it alwready told it to unpause? we start the domian in the pased state. i wonder if the qemu monitor has hung | |
| 17:14:14 | mriedem | fun | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack libvirt-guests.sh[18879]: Timeout expired while shutting down domains | |
| 17:14:14 | mriedem | Jul 25 17:13:36 devstack systemd[1]: libvirt-guests.service: Control process exited, code=exited status=1 | |
| 17:14:21 | mriedem | trying to restart libvirt-guests | |
| 17:15:19 | sean-k-mooney | is this a clean install of ubuntu 18.04? | |
| 17:15:37 | mriedem | well from a vexxhost image of 18.04 but yeah | |
| 17:16:15 | sean-k-mooney | strange i personlly havent had any issue with 18.04 i did an install fiday | |
| 17:16:33 | mriedem | me neither | |
| 17:17:57 | sean-k-mooney | are the vexhost image available for download | |
| 17:18:06 | mriedem | idk, i'm trashing this vm | |
| 17:19:12 | mnaser | the vexxhost images are straight up the ones shipped by ubuntu | |
| 17:19:12 | sean-k-mooney | ya i would just start over too to be honest. i suspect its somehitng to doe with libvirt/qemu or maybe apparmor but i would start clean | |
| 17:19:27 | sean-k-mooney | mnaser: the cloud images | |
| 17:19:32 | mnaser | yep | |
| 17:19:38 | mnaser | only thign we do is convert from qemu to raw | |
| 17:19:41 | mnaser | and upload | |
| 17:19:53 | sean-k-mooney | im guessing ye are using ceph as a backend then | |
| 17:20:12 | mnaser | indeed :) | |
| 17:20:13 | kashyap | On Fedora, the 'libvirt-guests' thing isn't even enabled: | |
| 17:20:15 | kashyap | $> systemctl status libvirt-guests | |
| 17:20:15 | kashyap | ● libvirt-guests.service - Suspend/Resume Running libvirt Guests | |
| 17:20:15 | kashyap | Loaded: loaded (/usr/lib/systemd/system/libvirt-guests.service; disabled; vendor preset: disabled) | |
| 17:20:18 | kashyap | Active: inactive (dead) | |
| 17:20:21 | kashyap | ... | |
| 17:20:46 | kashyap | But yeah, that timeout of 'libvirt-guests' looks spurious enough, might as well start over. | |
| 17:21:05 | sean-k-mooney | kashyap: would that not cause filesystem curruption if you did not suspend them on rebooting the host | |
| 17:21:06 | kashyap | (Also, not sure if that paused instance's QEMU process went 'defunct') | |
| 17:21:37 | efried | following up re os-vif and python-novaclient releases: Libs are required to do one release per milestone. os-vif was last released at m1, so we can expect the release team to propose that one. python-novaclient was released a couple weeks ago, so we're probably good on that one. | |
| 17:21:41 | sean-k-mooney | kashyap: im guessing the qemu moniotr process stoped processing messages form libvirt | |
| 17:22:47 | sean-k-mooney | efried: ok there is one think i would like to fix soonish but im only starting on it today | |
| 17:22:48 | kashyap | Yeah, but that doesn't tell us why. It could be any no. of reasons | |
| 17:23:33 | sean-k-mooney | kashyap: yep its proably quicker to kill it and spin up a clean vm | |
| 17:23:45 | sean-k-mooney | if mriedem hits it again we can take another look | |
| 17:24:22 | kashyap | sean-k-mooney: Yeah, on FS corruption, possibly "enterprise distros" would enable it | |
| 17:24:56 | sean-k-mooney | it look like ubuntu just enables it by defualt to be safe by defualt | |
| 17:31:23 | kashyap | sean-k-mooney: RHEL doesn't either, BTW. And one can configure what action 'libvirt-guests' can take on host shutdown | |
| 17:31:45 | sean-k-mooney | ok | |
| 17:32:01 | sean-k-mooney | well that is not related to the issue mriedem was having | |
| 17:32:17 | sean-k-mooney | the issue he was having was that the vm hung | |
| 17:32:49 | sean-k-mooney | and then the linux-guests scipt also hugn on shutdown for the same reason | |