| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-10-06 | |||
| 00:15:19 | mriedem | seems like the only thing available to me is the ansible env vars from the d-g setup in http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/devstack-gate-setup-host.txt.gz | |
| 00:15:27 | mtreinish | mriedem: local.conf should be sourced by grenade already: https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 | |
| 00:15:29 | mriedem | so i can get like GRENADE_OLD_BRANCH | |
| 00:16:56 | mriedem | yeah but this is a script running in the post_test_hook | |
| 00:17:01 | mriedem | would that carry over? | |
| 00:18:31 | mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/console.html#_2017-10-04_14_36_43_557726 | |
| 00:18:35 | mtreinish | ah, ok. Yeah it won't be sourced there | |
| 00:18:45 | melwitt | mtreinish: still failing after installing python3-rados, even though it shows up in py3 pip freeze on the job. was not expecting that. and when I try locally I can import rados in py3 http://logs.openstack.org/63/509663/3/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/2361fe2/logs/screen-g-api.txt.gz?level=ERROR | |
| 00:18:51 | mriedem | ok, so i need to do https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 from within the script | |
| 00:19:23 | melwitt | back to the drawing board | |
| 00:20:34 | mtreinish | melwitt: what about rbd on py3. Looking at the glance_store code it if either rados or rbd isn't present it'll set both to None | |
| 00:20:49 | mtreinish | melwitt: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L37-L42 | |
| 00:21:08 | melwitt | mtreinish: hm, good point. lemme go down that road | |
| 00:21:35 | melwitt | yeah you're right | |
| 00:21:38 | melwitt | thanks | |
| 00:21:51 | melwitt | need to get py3 rbd separately | |
| 00:21:56 | mtreinish | sure, np | |
| 00:22:10 | mtreinish | mriedem: probably | |
| 00:22:30 | mtreinish | mriedem: or something like that to source the vars you need from the conf file | |
| 00:29:32 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | |
| 00:29:33 | mriedem | mtreinish: check out this crazy shit ^ | |
| 02:41:19 | vivsoni | i am trying to attach FC, then attach ISCSI volume to nova instance.. while attaching it create a proper vlun entries in /dev/disk/by-pth/ | |
| 02:41:55 | vivsoni | but after detaching the iscsi volume some of the lun entries are instact in /dev/disk/by-path directory | |
| 02:44:11 | vivsoni | it would be great if someone can point me to code, from where the lun entries are created in /dev/disk/by-path | |
| 02:45:28 | vivsoni | i suspect some 'iscsiadm' or similar 'rescan' command is triggered and due to which the lun entries are create in /dev/disk/by-path... please correct me if i am worng | |
| 02:45:33 | vivsoni | thanks !!! | |
| 03:37:05 | mriedem | jaypipes: dansmith: i did it! http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | |
| 03:37:23 | mriedem | reproduced that 409 during claim resources in the scheduler, creating 1000 instances at once | |
| 03:40:09 | mriedem | retried 18 times across that 1000 | |
| 03:40:15 | mriedem | one of those poor saps just couldn't hack it | |
| 03:40:26 | mriedem | need to run with https://review.openstack.org/#/c/507705/2/nova/scheduler/client/report.py to find out which one i guess | |
| 03:41:43 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/509961 | |
| 04:43:59 | openstackgerrit | melanie witt proposed openstack/nova master: Improve the CellDatabases test fixture and usage https://review.openstack.org/508432 | |
| 04:44:00 | openstackgerrit | melanie witt proposed openstack/nova master: Elevate existing RequestContext to get bandwidth usage https://review.openstack.org/509968 | |
| 04:44:00 | openstackgerrit | melanie witt proposed openstack/nova master: Target context for build notification in conductor https://review.openstack.org/509967 | |
| 05:11:37 | hanish | one of my compute node is disabled due to 10 vm launch failing, how can i recover that node | |
| 05:20:37 | Tengu | you must re-enable nova agent, hanish | |
| 05:25:43 | hanish | @Tengu: i restarted nova-compute agent on compute node, but still i facing the issue | |
| 05:29:10 | Tengu | hmmm nope, not via systemctl | |
| 05:29:26 | Tengu | there's an openstack command for that, in order to re-enable it at openstack level | |
| 05:30:33 | takashin | ||
| 05:32:09 | Tengu | hanish: there's something like nova service-list - you should see it's disabled for your host. | |
| 05:32:17 | Tengu | then you have nova service-enable | |
| 05:32:30 | Tengu | hanish: I don't remember the "openstack unified" command for those. | |
| 05:38:30 | hanish | tengu: thanks | |
| 06:17:34 | Tengu | hanish: did it do the trick? | |
| 06:20:20 | hanish | Tengu: thanks, yes it worked. | |
| 06:23:26 | Tengu | hanish: good :). | |
| 06:23:51 | Tengu | I had that kind of issue earlier with tripleO. | |
| 06:36:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | |
| 06:50:21 | takashin | gmann: cdent: I have modified the spec for "Abort cold migration" function. Would you review https://review.openstack.org/#/c/334732/ again? | |
| 06:50:27 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 08:24:00 | zioproto | hello :) Working on Newton I have a region in my cloud where openstack usage list goes in stacktrace. Instance not found | |
| 08:24:23 | zioproto | looking at the nova bugs I did not find anything usefull | |
| 08:25:34 | zioproto | tracking my logs it looks like this is broken since I upgraded to Newton | |
| 08:25:57 | zioproto | dansmith: usually you like these database stories :) | |
| 08:26:53 | bauwser | zioproto: stacktrace ? | |
| 08:27:20 | bauwser | zioproto: by Newton, we began to use the API DB | |
| 08:27:45 | zioproto | https://pastebin.com/gtJbutvi | |
| 08:28:01 | zioproto | it is funny because this happens only in 1 production region | |
| 08:28:23 | zioproto | I have the same setup on dev/staging/prod and there it works | |
| 08:28:38 | zioproto | my feeling is that there is a broken database entry, or something specific to that region that breaks it | |
| 08:29:28 | zioproto | of course openstack server show 72afef44-1a4b-46e5-8cbc-7bd4f0eb31ff gives me a instance not found as well | |
| 08:29:39 | zioproto | what other tables should I dig to look for this uuid ? | |
| 08:32:06 | gmann | takashin: thanks. i will check soon. | |
| 08:33:13 | takashin | gmann: Thanks in advance. | |
| 08:39:47 | bauwser | zioproto: are you aware of those commands when you deploy a Newton cloud ? https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2 | |
| 08:40:25 | zioproto | bauwser: I think I did this when I upgraded to Mitaka | |
| 08:40:38 | zioproto | to split the nova db into two DBs | |
| 08:41:35 | bauwser | zioproto: if I were you, I'd be looking at the nova_api DB for the instance_mappings table | |
| 08:41:51 | zioproto | I go have a look | |
| 08:41:51 | bauwser | zioproto: and check which cell is for the instance UUID | |
| 08:42:23 | bauwser | zioproto: then looking at host_mappings if we have a cell record for each host in it | |
| 08:52:43 | zioproto | bauwser: my host_mappings table is empty, is that a bad sign ? | |
| 08:53:18 | bauwser | zioproto: indeed, how many hosts do you have? | |
| 08:53:31 | zioproto | you mean compute hosts ? | |
| 08:53:44 | zioproto | like hundreds | |
| 08:56:15 | zioproto | looks like we upgraded to newton without doing this cells housekeeping | |
| 08:56:32 | zioproto | this was mandatory at this point ? | |
| 08:56:38 | zioproto | to have a cell0 database ? | |
| 09:01:57 | bauwser | zioproto: I don't exactly remember when we used the api DB for getting the instance list | |
| 09:02:41 | bauwser | zioproto: but you should definitely map the hosts | |
| 09:06:18 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 09:07:53 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 09:15:11 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 10:35:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | |
| 10:35:29 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add sample test for instance audit https://review.openstack.org/480955 | |
| 11:03:52 | openstackgerrit | Merged openstack/nova master: Blacklist test_extend_attached_volume from cells v1 job https://review.openstack.org/509907 | |
| 11:17:57 | gibi | hm https://review.openstack.org/509907 has been merged so I guess it is recheck time | |
| 11:25:22 | zioproto | bauwser: in nova-manage cell_v2 simple_cell_setup [--transport-url <transport_url>] how does <transport_url> look like ? | |
| 11:28:58 | zioproto | ok I found it here https://docs.openstack.org/nova/latest/user/cells.html | |
| 11:43:33 | fried_rice | jaypipes yt? Wanted to brainstorm a couple of edge cases. | |
| 12:06:07 | gibi | could a second core look at this code removal patch? https://review.openstack.org/#/c/505164/ | |
| 12:18:33 | gibi | bauwser: hi! you can still support my embarrassment in https://review.openstack.org/#/c/509750 if you would like to :) | |
| 12:19:14 | fried_rice | Gluten free, if you please. | |
| 12:20:11 | fried_rice | cdent Perhaps you'd be willing to give some feedback on these edge cases that came to me in my sleep (or lack thereof) last night. | |
| 12:21:10 | cdent | I can try, but these cases keep confusing me, but since it will probably help to talk about it, shoot. | |
| 12:21:26 | fried_rice | If I ask for CUSTOM_FOO:4, is it *ever* legal for placement to give me 3 CUSTOM_FOOs from one RP and 1 from a different RP? | |
| 12:22:32 | cdent | In the fundament case of resource providers and inventory, no | |
| 12:22:40 | cdent | the request is a for a chunk of size 4 | |