| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-13 | |||
| 13:33:32 | mriedem | cdent: ok i figured out where that weird same-host allocation goofiness happens | |
| 13:34:26 | cdent | mriedem: sorry, only pay half attention, doing my taxes, which was a surprise gift for today | |
| 13:34:41 | mriedem | that is a fun gift | |
| 13:34:50 | mriedem | anyway, the allocations thing was a red herring | |
| 13:35:45 | mriedem | during live migration we call the scheduler for dest hosts and then do some checks on them in the conductor task, like the hypervisor version comparison between the source and dest, | |
| 13:35:56 | mriedem | if that fails, we call https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1527 to remove the allocations created on the dest provider by the scheduler, | |
| 13:36:13 | mriedem | if we get to https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1593 it assumes we're in a same-host resize situation, which is wrong | |
| 13:36:32 | mriedem | we end up sending {} for resource allocatoins and placement pukes on that https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1620 | |
| 13:36:54 | mriedem | we log a warning, and b/c we're doing this from an except block, oslo.log is also logging the original exception DestinationHypervisorTooOld - which makes it all very confusing | |
| 13:38:04 | frickler | mriedem: https://opendev.org/openstack-dev/devstack/src/branch/stable/queens/tools/fixup_stuff.sh#L86 | |
| 13:38:24 | frickler | seems ENABLE_VOLUME_MULTIATTACH is only set on controller and not on compute1 | |
| 13:39:11 | mriedem | frickler: gah that would do it | |
| 14:00:16 | gmann | mriedem: is it? | |
| 14:05:49 | mriedem | gmann: yeah https://bugs.launchpad.net/tempest/+bug/1819910 | |
| 14:05:50 | openstack | Launchpad bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,Triaged] | |
| 14:09:07 | mriedem | gmann: i'll try pushing a fix to tempest | |
| 14:11:23 | gmann | mriedem: thanks, is UCA thing will fix it. | |
| 14:14:10 | mriedem | gmann: ok i was working a patch but are you doing it now? | |
| 14:15:56 | gmann | mriedem: no, i am doing legacy job migration as of now. if you are working please push. | |
| 14:20:44 | mriedem | https://review.openstack.org/643052 | |
| 14:20:58 | mriedem | i'll push a nova stable/queens change that depends on ^ to make sure it works | |
| 14:28:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix live migration _remove_host_allocations warnings https://review.openstack.org/643055 | |
| 14:36:59 | openstackgerrit | Corey Bryant proposed openstack/nova master: add python 3.7 unit test job https://review.openstack.org/610694 | |
| 14:38:49 | brtknr | hello! | |
| 14:39:09 | brtknr | i have an instance stuck in BUILD which I do not seem to be able to delete... | |
| 14:39:23 | brtknr | whats the remedy | |
| 14:39:50 | brtknr | it appears when i do `openstack server list` but when i run `openstack server delete UUID`, i get: No server with a name or ID of '1a141848-c91a-450b-b4ad-6d2f81e4125e' exists. | |
| 14:40:37 | mriedem | brtknr: hmm, is there an entry for that instance_uuid in the nova_api.build_requests table? | |
| 14:41:00 | brtknr | in the database? | |
| 14:41:09 | brtknr | probably... | |
| 14:41:22 | brtknr | it exists as an entity somewhere... | |
| 14:41:26 | brtknr | but its not deletable | |
| 14:41:45 | mriedem | can you actually check the nova_api.build_requests table for a record with that instance_uuid? | |
| 14:41:52 | mriedem | because it should be temporary | |
| 14:43:51 | brtknr | mriedem: im checking, one sec | |
| 14:44:58 | brtknr | mysql> select * from build_requests; | |
| 14:44:59 | brtknr | Empty set (0.00 sec) | |
| 14:46:49 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: DNM: testing tempest-slow fix for subnode config https://review.openstack.org/643059 | |
| 14:47:37 | brtknr | mriedem: ive just deleted another server and its gone into the same state... still shows up under server list but cannot be deleted... | |
| 14:48:42 | mriedem | brtknr: so it's probably in a cell but the instance_mappings record in the nova_api database for it does not have cell_id set for some reason | |
| 14:48:51 | mriedem | check the nova_api.instance_mappings table for the instance_uuid in question, | |
| 14:49:01 | mriedem | the cell_id column should be set, but i'm guessing it's not in your case | |
| 14:49:44 | mriedem | that would suggest something is failing in your nova-conductor service during scheduling where the instance mapping is not updated to point at the cell in which the instance record actually exists | |
| 14:50:01 | mriedem | which is i'm guessing either cell0 or the 'nova' database (cell1) for you | |
| 14:57:20 | openstackgerrit | Merged openstack/nova stable/pike: [pike-only] Fix resize_instance rpcapi call https://review.openstack.org/603439 | |
| 14:57:27 | openstackgerrit | Merged openstack/nova stable/pike: tox: Don't write byte code (maybe) https://review.openstack.org/636919 | |
| 14:57:36 | openstackgerrit | Merged openstack/nova stable/pike: PCI: do not force remove allocated devices https://review.openstack.org/635074 | |
| 14:57:46 | openstackgerrit | Merged openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | |
| 14:58:47 | dansmith | wow, pike | |
| 15:15:10 | mriedem | dansmith: if you like those, mayhap you want to hit these other rocky +2'ed changes https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+label:Code-Review=2 | |
| 15:16:25 | dansmith | no good deed | |
| 15:23:07 | openstackgerrit | Merged openstack/nova stable/rocky: Handle templated cell mappings in nova-status https://review.openstack.org/642611 | |
| 15:26:14 | dansmith | mriedem: I got stopped yesterday with that bigger ironic change at the top there | |
| 15:27:06 | mriedem | just plow right through those concerns | |
| 15:29:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance to make place for port healing https://review.openstack.org/637953 | |
| 15:29:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations https://review.openstack.org/637955 | |
| 15:29:34 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance (2) https://review.openstack.org/637954 | |
| 15:29:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: cache neutron ports in heal allocation https://review.openstack.org/638207 | |
| 15:43:00 | openstackgerrit | Merged openstack/nova master: Remove duplicate cleanup in functional tests https://review.openstack.org/636996 | |
| 15:47:38 | mriedem | gibi: i went over the docs for bw providers https://review.openstack.org/#/c/642064/ some comments inline | |
| 15:47:46 | mriedem | let me know if you want to discuss my comments | |
| 15:47:52 | gibi | mriedem: thanks. looking | |
| 15:49:41 | gmann | melwitt: mriedem nova-next is failing with same reason - http://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/screen-n-cpu.txt.gz#_Mar_13_14_31_14_886123 | |
| 15:51:26 | mriedem | gmann: i don't think that's the same, | |
| 15:51:29 | mriedem | and i don't think nova-next is multinode | |
| 15:52:06 | mriedem | http://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/libvirt/qemu/instance-00000001.txt.gz | |
| 15:52:11 | gmann | mriedem: not same as tempest-slow. i mean cert issue which was occurring early starting from when migrating it to bionic | |
| 15:52:13 | mriedem | -vnc 127.0.0.1:0,tls,x509verify=/etc/pki/libvirt-vnc: Failed to start VNC server: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file. | |
| 15:52:21 | gmann | yeah this one. | |
| 15:52:42 | gmann | melwitt: was trying it with 1024 bit. | |
| 15:53:02 | mriedem | ok i don't know about the vnc / cert stuff | |
| 15:53:21 | gmann | mriedem: sorry for mixing it. it is for https://review.openstack.org/#/c/639017/ | |
| 15:55:37 | cfriesen | I think this should be a fairly uncontroversial bugfix (has +2 from efried): https://review.openstack.org/#/c/641480/ | |
| 15:58:26 | mriedem | dansmith: you'll never believe it by the $11 lady won | |
| 15:58:29 | mriedem | *but the | |
| 15:58:36 | mriedem | i've been carole'ed | |
| 15:58:47 | dansmith | mriedem: now you have to explain to maya why she's not an idiot after all? | |
| 15:58:56 | mriedem | i think she's sleeping | |
| 15:59:12 | dansmith | heh | |
| 16:00:28 | mriedem | too much colonial penn commercials | |
| 16:00:40 | mriedem | and oxygen tanks getting you down | |
| 16:01:16 | dansmith | is the oatmeal guy still double-dipping on insurance commercials? | |
| 16:02:12 | dansmith | oh I guess colonial penn was someone else | |
| 16:02:18 | mriedem | alex trebek | |
| 16:02:20 | mriedem | RIP | |
| 16:02:29 | mriedem | oatmeal was wilford brimley | |
| 16:02:32 | mriedem | from "the thing" fame | |
| 16:03:01 | mriedem | https://www.youtube.com/watch?v=3qSbYEbtfoY | |
| 16:03:05 | dansmith | yeah but he also did Liberty Mutual insurance, another daytime TV staple | |
| 16:03:07 | mriedem | here is now doing heavy computer work | |
| 16:04:52 | mriedem | note the (1) empty bottle of vodka on the desk and (2) .38 special in the drawer | |
| 16:05:21 | dansmith | I have never seen this movie, but the highly accurate 1980s computer simulation makes me think I'm missing out | |
| 16:05:28 | mriedem | i think that's a 38 special anyway, it's been awhile since i've listened to lynard skynard | |
| 16:05:37 | mriedem | the thing is one of my all time favorites | |
| 16:06:10 | mriedem | the 80s version of course | |
| 16:06:10 | dansmith | I don't recall them providing details of .38 special identification | |
| 16:06:41 | mriedem | i guess i'm thinking of saturday night special | |
| 16:06:51 | dansmith | I believe there was something about technique.. perhaps holding on loosely.. but that wasn't skynard | |
| 16:06:56 | mriedem | are those equivalent hand guns? | |
| 16:07:21 | mriedem | nobody likes the actual band "38 special" | |