| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-17 | |||
| 15:26:39 | SteelyDan | but even post pike it will be for deleted instances only, | |
| 15:26:47 | SteelyDan | so I'm not positive why this was even a thing on pike | |
| 15:27:48 | melwitt | what do you mean? the compute node will remove allocations for a local deleted instance on pike? | |
| 15:27:57 | melwitt | via auto-heal or something? | |
| 15:28:08 | SteelyDan | yes, | |
| 15:28:13 | SteelyDan | it used to always auto-heal everything, | |
| 15:28:24 | SteelyDan | but as of pike it should only do it for deleted instances once all the ocata computes are gone | |
| 15:28:25 | cdent | looking a bit more closely, it seems the real issue on the internal discussion is this part of the bug fix: https://review.openstack.org/#/c/580499/ , for cleaning up host mappings | |
| 15:28:27 | SteelyDan | let me see if I an find that | |
| 15:28:31 | melwitt | I see | |
| 15:28:56 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: add zvm into support matrix https://review.openstack.org/593079 | |
| 15:28:57 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add zvm admin intro and hypervisor information https://review.openstack.org/593080 | |
| 15:28:58 | openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add zvm CI information https://review.openstack.org/593081 | |
| 15:29:18 | SteelyDan | melwitt: https://github.com/openstack/nova/blob/stable/pike/nova/compute/resource_tracker.py#L1249-L1258 | |
| 15:29:37 | SteelyDan | melwitt: prior to pike, the other conditions in that heal/check loop would heal things, but as of pike only the instance.deleted one would | |
| 15:30:05 | SteelyDan | because as of pike the scheduler/conductor/api are authoritative for allocations | |
| 15:30:09 | melwitt | a-ha, thanks | |
| 15:31:54 | cdent | le sigh. it seems the root cause of the internal problem is: we messed various database tables and now things don't look like they should | |
| 15:32:14 | cdent | doctor it hurts when I punch myself! | |
| 15:35:41 | fried_rice | Choke yourself instead. Self-limiting. | |
| 15:39:07 | cdent | what you do in the privacy of your own home... | |
| 15:40:32 | SteelyDan | hehe, thanks I was looking for how to acknowledge the kink | |
| 15:49:09 | cdent | said SteelyDan | |
| 15:49:26 | SteelyDan | :) | |
| 16:35:34 | SteelyDan | sean-k-mooney: did you see my query here earlier? | |
| 16:35:54 | sean-k-mooney | oh am no ill scoll back | |
| 16:36:13 | SteelyDan | (repeat) sean-k-mooney: you confirmed that the new approach works for LB right? If so, can you comment on this patch? https://review.openstack.org/#/c/591735 | |
| 16:37:13 | sean-k-mooney | i confimed that the current code on master works. this is the queens backport of the same? | |
| 16:37:28 | SteelyDan | sean-k-mooney: you don't need to confirm it separately for queens | |
| 16:37:41 | SteelyDan | but if you could comment there I'd appreciate it | |
| 16:38:10 | sean-k-mooney | yes ill just triple check i have the commit in question on the relevent nodes | |
| 16:38:44 | SteelyDan | I think we've already established when the event is sent from neutron in the LB case, so it really shouldn't even be an issue, but... | |
| 16:39:53 | sean-k-mooney | yep i have the master change on both linux bidge node an the conf change. ill comment to that effect | |
| 16:47:18 | sean-k-mooney | speaking of LM testing i should work on that email to the list. | |
| 16:57:16 | melwitt | SteelyDan: docs changes for rc2 proposed starting here https://review.openstack.org/593079 if you could pls take a look | |
| 16:57:39 | melwitt | backports | |
| 16:57:42 | SteelyDan | ugh okay, I'll add it to the list | |
| 16:57:58 | melwitt | thanks | |
| 17:11:27 | SteelyDan | melwitt: don't we know that zvm doesn't support rebuild? | |
| 17:14:33 | melwitt | SteelyDan: mriedem asked them to change it from missing => unknown because since they support spawn and destroy, it sounds like it could work https://review.openstack.org/#/c/532720/8/doc/source/user/support-matrix.ini@378 | |
| 17:15:28 | melwitt | but they aren't testing that path, so "unknown" | |
| 17:17:51 | SteelyDan | it's not implemented in the driver, and untested, which sounds a lot more like "unsupported" than "maybe it'll work if code we wrote long ago to handle it for normal hypervisors, which zvm is not" | |
| 17:17:53 | SteelyDan | but.. okay | |
| 17:18:39 | melwitt | yeah, I think that's why they put "missing" initially | |
| 17:18:48 | SteelyDan | ...yeah | |
| 17:18:53 | SteelyDan | so glad we added this | |
| 17:19:00 | SteelyDan | so are we looking for fast approvals on these I assume? | |
| 17:19:09 | SteelyDan | or do you want to wait for another stabler on monday? | |
| 17:19:57 | melwitt | I was thinking mriedem could get them on monday | |
| 17:20:43 | melwitt | I wanted them to be ready to go. I wasn't sure if we need to fast approve | |
| 17:20:57 | melwitt | I guess I could ask smcginnis | |
| 17:21:49 | SteelyDan | you know better than I at this point when rc2 will be, which is why I'm asking | |
| 17:22:12 | melwitt | final RCs are due Aug 23, so AFAIK we are fine until then | |
| 17:37:05 | SteelyDan | ack | |
| 18:08:37 | smcginnis | melwitt: The question was whether to do another RC? | |
| 18:14:44 | melwitt | smcginnis: no, question was, should we be looking at fast-approving doc backports to stable/rocky for RC2 today (other stable cores are not around) or is it fine to wait until monday? | |
| 18:15:04 | melwitt | my understanding is final RCs are due Aug 23, so there isn't a rush. but please correct me if I'm wrong | |
| 18:18:55 | smcginnis | melwitt: That's correct. You have until next Thursday if there is a need to do another RC. | |
| 18:19:26 | melwitt | ok thank you | |
| 18:19:32 | smcginnis | melwitt: So if there isn't a need to get that proposed RC2 out right away, and you know there's likely to be more, it might make sense to hold off on that for now and just do the RC2 next week. | |
| 18:19:59 | smcginnis | melwitt: But if there is a need to get the commits since RC1 out there right away, we can definitely do RC2 now and another RC3 next week. | |
| 18:20:15 | smcginnis | In general we just like to minimize the number of RCs, but practically it doesn't really matter too much. | |
| 18:20:48 | melwitt | I see. thanks, that's helpful | |
| 18:24:05 | mnaser | super unrelated, but have you seen 'sh: write error: No space left on device' in console logs for cirros in tempest much? | |
| 18:24:30 | mnaser | i'm having problems with the (i think) last issue to get centos passing on openstack ansible, it's failing to ssh to the floating ip with a timeout .. http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/aio1-utility/stestr_results.html | |
| 18:24:54 | mnaser | the console logs seem to show that it fails to add teh default route and out of space writing things on disk | |
| 18:26:28 | melwitt | anecdotally, I've run out of disk space using ubuntu cloud images with devstack + ceph because the devstack-plugin-ceph defaults to 8GB disk space and the ubuntu xenial cloud image comes with 10GB disk space by default | |
| 18:26:52 | melwitt | not sure where you're getting your images from but it could be something like that? I dunno | |
| 18:27:22 | mnaser | we're using cirros qcow2 images, so i think it shouldn't be too wildly different | |
| 18:27:29 | mnaser | xenial works fine which makes me a bit curious | |
| 18:28:18 | melwitt | oh, right I see, you're saying image is same, host OS is different | |
| 18:28:29 | mnaser | yes | |
| 18:28:32 | melwitt | yeah, I haven't heard about that before | |
| 18:29:02 | mnaser | i can see the floating ip getting assigned successfully too, ill dig in some more | |
| 18:32:50 | openstackgerrit | Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698 | |
| 18:32:51 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |
| 18:33:50 | SteelyDan | melwitt: ^ split that patch and also put ideas for how to configure this in the second one | |
| 18:33:56 | SteelyDan | which you might want to opine on | |
| 18:36:42 | melwitt | coolness, adding to my list | |
| 19:06:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Batch results per cell when doing cross-cell listing https://review.openstack.org/592698 | |
| 19:06:18 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Make instance_list perform per-cell batching https://review.openstack.org/593131 | |
| 20:17:26 | openstackgerrit | Joshua Harlow proposed openstack/nova master: Add exact match aggregate image properties matcher/filter https://review.openstack.org/593167 | |
| 20:18:07 | openstackgerrit | Joshua Harlow proposed openstack/nova master: Add exact match aggregate image properties matcher/filter https://review.openstack.org/593167 | |
| 20:18:50 | openstackgerrit | Joshua Harlow proposed openstack/nova master: Add exact match aggregate image properties matcher/filter https://review.openstack.org/593167 | |
| 20:27:32 | openstackgerrit | Joshua Harlow proposed openstack/nova master: Add exact match aggregate image properties matcher/filter https://review.openstack.org/593167 | |
| 20:51:38 | needssleep | Out of curiosity, have any changes merged to resource tracking in say the last couple days? | |
| 20:51:44 | needssleep | on the current state of master branch | |
| 20:51:53 | needssleep | http://logs.openstack.org/87/592387/1/check/ironic-inspector-tempest-dsvm-python3/3aba945/logs/screen-n-cpu.txt.gz#_Aug_16_14_31_27_527100 is why I'm curious | |
| 20:57:34 | openstackgerrit | Joshua Harlow proposed openstack/nova master: Add exact match aggregate image properties matcher/filter https://review.openstack.org/593167 | |
| 21:25:21 | openstackgerrit | Dmitry Sutyagin proposed openstack/nova-specs master: Allow disabling KSM / mem-merge via extra spec https://review.openstack.org/593197 | |
| 21:36:14 | mnaser | ok, managed to resolve that first issue but now | |
| 21:36:24 | mnaser | http://logs.openstack.org/78/590778/7/gate/openstack-ansible-deploy-aio_metal-opensuse-423/a4388cf/logs/openstack/aio1-nova/nova-compute.log.txt.gz (warning, large log file as usual) | |
| 21:36:42 | mnaser | the instance gets setup, started paused, resumes (twice?) and pauses again | |
| 21:37:04 | mnaser | afaik.. normal behaviour is that it just goes paused => resumed ? | |
| 21:37:44 | mnaser | instance id 51f86e01-8372-4a6f-b9aa-4af75b6d9bf8 if it helps looking at logs | |
| 21:37:54 | mnaser | http://logs.openstack.org/78/590778/7/gate/openstack-ansible-deploy-aio_metal-opensuse-423/a4388cf/logs/openstack/aio1-nova/nova-compute.log.txt.gz#_2018-08-17_21_05_55_514 | |
| 21:41:03 | mnaser | well, libvirt logs show it going from paused to resumed to paused http://paste.openstack.org/show/728337/ | |
| 21:41:44 | mnaser | http://logs.openstack.org/78/590778/7/gate/openstack-ansible-deploy-aio_metal-opensuse-423/a4388cf/logs/host/libvirt/qemu/instance-00000001.log.txt.gz hmm | |
| 23:59:58 | openstackgerrit | Merged openstack/nova master: comment correction for libvirt multiattach https://review.openstack.org/593050 | |
| #openstack-nova - 2018-08-18 | |||
| 00:02:35 | melwitt | needssleep: no, nothing since a couple of weeks ago. this only started the last couple of days? | |