| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-05 | |||
| 23:14:12 | mriedem | and 2 of the events don't have project ids | |
| 23:14:15 | mriedem | i bet i know what this is | |
| 23:14:22 | mriedem | sync_instance_power_state periodic | |
| 23:14:45 | mriedem | the last 2 are 10 minutes apart | |
| 23:14:51 | mriedem | which is the default on that periodic | |
| 23:16:07 | mriedem | ha, yeah, because the fake driver that i'm using doesn't change the state on the fake guest when you power it off | |
| 23:16:14 | mriedem | so the db says it's stopped but the "hypervisor" doesn't | |
| 23:16:16 | mriedem | so nova stops it | |
| 23:16:24 | mriedem | sev1 fake driver bug | |
| 23:18:57 | mtreinish | mriedem: hah, nice | |
| 23:21:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement power_off/power_on for the FakeDriver https://review.openstack.org/509935 | |
| 23:38:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add live.migration.force.complete to the legacy notification whitelist https://review.openstack.org/508877 | |
| 23:38:22 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional migrate force_complete test https://review.openstack.org/509924 | |
| 23:38:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional for live migrate delete https://review.openstack.org/509925 | |
| 23:38:23 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Remove dest node allocations during live migration rollback https://review.openstack.org/509926 | |
| #openstack-nova - 2017-10-06 | |||
| 00:09:35 | mriedem | mtreinish: does sourcing /opt/stack/new/devstack/lib/nova in grenade not also source the /opt/stack/new/devstack/local.conf? | |
| 00:11:11 | mtreinish | mriedem: probably not, in devstack the assumption is likely that is sourced well before lib/nova is called | |
| 00:12:16 | mriedem | alright | |
| 00:12:54 | mriedem | hmm, so can i even source this thing from within a post_test_hook? | |
| 00:12:54 | mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/new/local.conf.txt.gz | |
| 00:13:05 | mriedem | i need to get the CELLSV2_SETUP value | |
| 00:15:19 | mriedem | seems like the only thing available to me is the ansible env vars from the d-g setup in http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/devstack-gate-setup-host.txt.gz | |
| 00:15:27 | mtreinish | mriedem: local.conf should be sourced by grenade already: https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 | |
| 00:15:29 | mriedem | so i can get like GRENADE_OLD_BRANCH | |
| 00:16:56 | mriedem | yeah but this is a script running in the post_test_hook | |
| 00:17:01 | mriedem | would that carry over? | |
| 00:18:31 | mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/console.html#_2017-10-04_14_36_43_557726 | |
| 00:18:35 | mtreinish | ah, ok. Yeah it won't be sourced there | |
| 00:18:45 | melwitt | mtreinish: still failing after installing python3-rados, even though it shows up in py3 pip freeze on the job. was not expecting that. and when I try locally I can import rados in py3 http://logs.openstack.org/63/509663/3/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/2361fe2/logs/screen-g-api.txt.gz?level=ERROR | |
| 00:18:51 | mriedem | ok, so i need to do https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 from within the script | |
| 00:19:23 | melwitt | back to the drawing board | |
| 00:20:34 | mtreinish | melwitt: what about rbd on py3. Looking at the glance_store code it if either rados or rbd isn't present it'll set both to None | |
| 00:20:49 | mtreinish | melwitt: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L37-L42 | |
| 00:21:08 | melwitt | mtreinish: hm, good point. lemme go down that road | |
| 00:21:35 | melwitt | yeah you're right | |
| 00:21:38 | melwitt | thanks | |
| 00:21:51 | melwitt | need to get py3 rbd separately | |
| 00:21:56 | mtreinish | sure, np | |
| 00:22:10 | mtreinish | mriedem: probably | |
| 00:22:30 | mtreinish | mriedem: or something like that to source the vars you need from the conf file | |
| 00:29:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | |
| 00:29:33 | mriedem | mtreinish: check out this crazy shit ^ | |
| 02:41:19 | vivsoni | i am trying to attach FC, then attach ISCSI volume to nova instance.. while attaching it create a proper vlun entries in /dev/disk/by-pth/ | |
| 02:41:55 | vivsoni | but after detaching the iscsi volume some of the lun entries are instact in /dev/disk/by-path directory | |
| 02:44:11 | vivsoni | it would be great if someone can point me to code, from where the lun entries are created in /dev/disk/by-path | |
| 02:45:28 | vivsoni | i suspect some 'iscsiadm' or similar 'rescan' command is triggered and due to which the lun entries are create in /dev/disk/by-path... please correct me if i am worng | |
| 02:45:33 | vivsoni | thanks !!! | |
| 03:37:05 | mriedem | jaypipes: dansmith: i did it! http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | |
| 03:37:23 | mriedem | reproduced that 409 during claim resources in the scheduler, creating 1000 instances at once | |
| 03:40:09 | mriedem | retried 18 times across that 1000 | |
| 03:40:15 | mriedem | one of those poor saps just couldn't hack it | |
| 03:40:26 | mriedem | need to run with https://review.openstack.org/#/c/507705/2/nova/scheduler/client/report.py to find out which one i guess | |
| 03:41:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/509961 | |
| 04:43:59 | openstackgerrit | melanie witt proposed openstack/nova master: Improve the CellDatabases test fixture and usage https://review.openstack.org/508432 | |
| 04:44:00 | openstackgerrit | melanie witt proposed openstack/nova master: Target context for build notification in conductor https://review.openstack.org/509967 | |
| 04:44:00 | openstackgerrit | melanie witt proposed openstack/nova master: Elevate existing RequestContext to get bandwidth usage https://review.openstack.org/509968 | |
| 05:11:37 | hanish | one of my compute node is disabled due to 10 vm launch failing, how can i recover that node | |
| 05:20:37 | Tengu | you must re-enable nova agent, hanish | |
| 05:25:43 | hanish | @Tengu: i restarted nova-compute agent on compute node, but still i facing the issue | |
| 05:29:10 | Tengu | hmmm nope, not via systemctl | |
| 05:29:26 | Tengu | there's an openstack command for that, in order to re-enable it at openstack level | |
| 05:30:33 | takashin | ||
| 05:32:09 | Tengu | hanish: there's something like nova service-list - you should see it's disabled for your host. | |
| 05:32:17 | Tengu | then you have nova service-enable | |
| 05:32:30 | Tengu | hanish: I don't remember the "openstack unified" command for those. | |
| 05:38:30 | hanish | tengu: thanks | |
| 06:17:34 | Tengu | hanish: did it do the trick? | |
| 06:20:20 | hanish | Tengu: thanks, yes it worked. | |
| 06:23:26 | Tengu | hanish: good :). | |
| 06:23:51 | Tengu | I had that kind of issue earlier with tripleO. | |
| 06:36:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | |
| 06:50:21 | takashin | gmann: cdent: I have modified the spec for "Abort cold migration" function. Would you review https://review.openstack.org/#/c/334732/ again? | |
| 06:50:27 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 08:24:00 | zioproto | hello :) Working on Newton I have a region in my cloud where openstack usage list goes in stacktrace. Instance not found | |
| 08:24:23 | zioproto | looking at the nova bugs I did not find anything usefull | |
| 08:25:34 | zioproto | tracking my logs it looks like this is broken since I upgraded to Newton | |
| 08:25:57 | zioproto | dansmith: usually you like these database stories :) | |
| 08:26:53 | bauwser | zioproto: stacktrace ? | |
| 08:27:20 | bauwser | zioproto: by Newton, we began to use the API DB | |
| 08:27:45 | zioproto | https://pastebin.com/gtJbutvi | |
| 08:28:01 | zioproto | it is funny because this happens only in 1 production region | |
| 08:28:23 | zioproto | I have the same setup on dev/staging/prod and there it works | |
| 08:28:38 | zioproto | my feeling is that there is a broken database entry, or something specific to that region that breaks it | |
| 08:29:28 | zioproto | of course openstack server show 72afef44-1a4b-46e5-8cbc-7bd4f0eb31ff gives me a instance not found as well | |
| 08:29:39 | zioproto | what other tables should I dig to look for this uuid ? | |
| 08:32:06 | gmann | takashin: thanks. i will check soon. | |
| 08:33:13 | takashin | gmann: Thanks in advance. | |
| 08:39:47 | bauwser | zioproto: are you aware of those commands when you deploy a Newton cloud ? https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2 | |
| 08:40:25 | zioproto | bauwser: I think I did this when I upgraded to Mitaka | |
| 08:40:38 | zioproto | to split the nova db into two DBs | |
| 08:41:35 | bauwser | zioproto: if I were you, I'd be looking at the nova_api DB for the instance_mappings table | |
| 08:41:51 | bauwser | zioproto: and check which cell is for the instance UUID | |
| 08:41:51 | zioproto | I go have a look | |
| 08:42:23 | bauwser | zioproto: then looking at host_mappings if we have a cell record for each host in it | |
| 08:52:43 | zioproto | bauwser: my host_mappings table is empty, is that a bad sign ? | |
| 08:53:18 | bauwser | zioproto: indeed, how many hosts do you have? | |
| 08:53:31 | zioproto | you mean compute hosts ? | |
| 08:53:44 | zioproto | like hundreds | |
| 08:56:15 | zioproto | looks like we upgraded to newton without doing this cells housekeeping | |
| 08:56:32 | zioproto | this was mandatory at this point ? | |